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Executive  summary 


Within  the  Department  of  Defense  (DoD),  large  multi-function  com¬ 
petitions1  are  a  popular  competitive  sourcing  approach  because  it  is 
believed  that  this  approach  generates  larger  savings  and  improved 
service  quality,  and  is  less  expensive  to  implement.  Moreover,  it  is  a 
way  to  meet  the  President’s  competitive  sourcing  goals  with  fewer 
competitions.  However,  there  is  also  the  perception  that  large  multi¬ 
function  competitions  limit  small  business  opportunities,  take  an 
excessive  amount  of  time,  and  cause  excessive  workforce  disruption. 

To  understand  the  role  of  large,  multi-function  competitions  in 
DoD’s  strategy  to  meet  the  President’s  Management  Agenda,  CNA 
conducted  a  study  of  the  costs,  savings,  and  performance  associated 
with  16  large  (greater  than  100  full-time  equivalents  (FTEs)  at 
announcement),  multi-function  and  multi-site  competitions  com¬ 
pleted  between  1996  and  2000.  The  purpose  of  the  study  is  to  assess 
the  degree  to  which  large-scale  competitions  completed  have 
resulted  in  increased  savings  and  service  quality  and  decreased  the 
costs  and  difficulty  of  conducting  the  competitions.  Specifically,  we 
attempted  to  identify  the  expected,  observed,  and  effective  savings; 
economies  of  scale;  opportunities  for  small  business;  ease  of  imple¬ 
mentation;  and  competition  process.  As  a  point  of  comparison,  we 
also  examined  several  multi-site,  single-function  competitions. 


1.  These  large-scale  competitions  are  frequently  referred  to  as  BOS  com¬ 
petitions  because  they  commonly  include  many  of  the  support  func¬ 
tions  at  an  installation  or  base.  Because  there  is  no  uniform  definition 
of  what  constitutes  a  BOS  competition,  we  examined  large  competitions 
(100  FTEs  or  more  at  announcement)  that  included  one  or  more  com¬ 
mercial  functions  or  activities. 
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Conclusion 


Based  on  our  review  of  16  large  multi-function/ site  competitions,  we 
have  concluded  that  large,  multi-function  or  multi-site  competitions 
can  play  a  significant  role  in  DoD’s  strategy  to  implement  its  compet¬ 
itive  sourcing  program.  Officials  believed  that  large,  multi-function/ 
site  competitions  provided  flexibility  for  innovation,  greater  savings, 
lower  overhead,  clear  command  and  control  accountability,  and 
faster,  less  costly  competitions. 

Large  competitions  typically  result  in  higher  savings 

The  16  competitions  we  reviewed  have  an  average  expected  savings  of 
48  percent,  or  15  percentage  points  higher  than  the  average  expected 
savings  rate  of  33  percent  for  the  population  of  competitions  in  the 

o 

CAMIS  database.  Although  our  analysis  of  observed  and  effective 
savings  rates  was  limited  to  11  of  the  16  competitions,  it  indicated  that 
the  average  savings  rates  were  44  and  49  percent,  respectively. 

Our  examination  of  the  16  competitions  and  the  population  of 
CAMIS  data  indicates  that  there  is  positive  correlation  between  com¬ 
petition  size  and  expected  savings.  There  is  a  slight  negative  impact 
on  savings  rates  as  additional  functions  are  added  (between  2.7  and 
4.5  percentage  points).  However,  these  negative  effects  are  out¬ 
weighed  by  the  savings  associated  with  increases  in  size  (see  figure  1) . 

Post-competition  performance  is  satisfactory 

Overall,  the  functional  managers,  contracting  representatives,  and 
customers  who  we  interviewed  considered  post-competition  perfor¬ 
mance  of  large  multi-function  and  multi-site  competitions  to  be  satis¬ 
factory.  Post-competition  performance  was  ranked  lowest  for 
competitions  that  were  in  their  first  or  second  year  of  operation 
because  some  transition  issues  were  still  unresolved  (see  figure  2) . 


2.  The  16  competitions  should  not  be  viewed  as  a  representative  sample  of 
the  population  of  DoD  competitions. 
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Figure  1 .  Relationship  between  expected  savings  and  competition  size 


Number  of  Billets 


Figure  2.  Distribution  of  competitions’  performance  by  number 
and  cost 
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Large  competitions  are  efficient  to  compete 

Our  sample  of  16  competitions  took  an  average  of  25  months  to  com¬ 
plete,  similar  to  the  27-month  average  of  other  A-76  competitions 
DoD  completed  during  roughly  the  same  time  period.  We  found  that 
the  16  competitions  were  well  within  the  statutory  time  limit  imposed 
on  DoD  competitions. 
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Small  business  opportunities  are  not  substantially  decreased  with 
large  multi-function  competitions 

About  38  percent,  or  6  competitions,  were  restricted  to  small  busi¬ 
nesses.  Three  additional  contracts  were  awarded  to  small  businesses 
even  though  the  competition  was  unrestricted.  Further,  those  con¬ 
tracts  awarded  to  large  firms  had  significant  small  business  goals. 
Small  business  contractors  indicated  that  large  competitions  can 
create  opportunities  for  improved  cash  flow,  mentoring,  and  business 
growth. 

Competition  size  is  only  one  factor  in  the  packaging  decision 

Our  review  of  the  16  competitions  indicated  that,  when  making  deci¬ 
sions  on  packaging,  officials  frequently  placed  a  higher  priority  on 
factors  that  were  internal  to  the  organization  such  as  meeting 
assigned  FTE  targets  or  retaining  the  original  organizational  struc¬ 
ture.  As  important  as  size  is  to  the  effectiveness  of  a  competition,  it 
should  be  balanced  against  the  goals  or  mission  of  the  organization, 
promote  effective  competition  by  considering  the  structure  and  com¬ 
position  of  the  private  sector,  and  be  realistic  with  relevant  labor 
market  and  economic  conditions. 

Large,  multi-function/site  competitions  increase  the  complexity 
and  coordination  needs  of  transition 

Although  conducting  large-scale  competitions  can  yield  great  effi¬ 
ciencies,  the  risk  of  non-performance  increases  as  the  number  of  ser¬ 
vice  providers  decreases.  Therefore,  it  is  vital  that  transitions  for  large 
competitions  be  well  planned  and  managed  by  both  the  government 
and  the  new  provider.  In  the  16  competitions  examined,  neither  the 
government  nor  the  contractor  placed  sufficient  emphasis  on  com¬ 
prehensive  transition  planning.  The  government’s  primary  focus  was 
on  placing  adversely  affected  employees,  often  at  the  expense  of  sus¬ 
taining  operations.  The  contractor’s  primary  focus  was  on  the  logis¬ 
tics  of  the  transition,  and  not  on  becoming  familiar  with  the 
operations  being  assumed. 
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Quality  assurance  needs  new  focus 

Quality  assurance  is  essential  to  accurately  measure  the  quality  of  the 
new  provider’s  performance.  In  our  review  of  the  16  competitions,  we 
found  that: 

•  QAEs  are  generally  performing  quality  control,  not  quality 
assurance  and  expect  the  provider  to  perform  much  the  same 
way  as  the  in-house  team  had  performed. 

•  QAEs  are  not  sufficiently  trained  prior  to  assuming  their  duties. 

•  Quality  issues  are  frequently  related  to  the  lack  of  fully  devel¬ 
oped  performance  metrics. 

•  In-house  performance  did  not  receive  the  same  scrutiny  as  did 
contractor  performance. 


Recommendations 

We  recommend  that  the  Department  consider  increasing  the 
number  of  large,  multi-function  or  multi-site  competitions  to 
increase  savings  and  accelerate  implementation  of  the  competitive 
sourcing  program.  However,  for  this  recommendation  to  be  most  suc¬ 
cessful,  the  Department  should  also: 

•  Ensure  that  the  scope  of  the  competition  supports  mission  goals,  indus¬ 
try  market  structure,  and  prevailing  labor  and  economic  conditions. 

•  Improve  the  packaging  process  by  examining  all  factors  that  affect  the 
quality  of  competition  and  level  of  savings. 

•  Improve  the  transition  to  the  new  provider  by  improving  the  quality  of 
government  and  contractor  transition  plans,  balancing  mission  accom¬ 
plishment  with  employee  placement,  matching  length  of  transition  to 
competition  size  and  complexity,  providing  training  to  those  monitoring 
the  new  provider,  and  establishing  the  monitoring  organization  before 
performance  starts. 

•  Improve  quality  assurance  by  focusing  on  output  or  results  rather  than 
process,  providing  monitors  with  adequate  training  including  contract/ 
MEO  requirements,  and  improving  performance  measures. 
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Organization  of  the  report 

We  begin  with  a  brief  introduction  that  provides  the  background  of 
the  study  and  summarizes  the  competitions.  Next,  we  present  our 
findings  on  the  questions  posed  to  us — questions  regarding  cost,  per¬ 
formance,  duration  of  the  competitions,  and  small  business  opportu¬ 
nities.  To  address  how  well  the  competition  process  worked,  we 
analyzed  each  step  of  the  process  used  by  the  16  competitions  to  iden¬ 
tify  its  contribution  to  their  success  or  lack  thereof.  Because  the  issues 
of  packaging,  transition,  and  quality  assurance  are  critical  factors  in 
the  decision  whether  to  conduct  large  or  small  competitions,  we 
present  each  of  these  steps  as  separate  sections.  Our  findings  on  the 
remaining  steps  in  the  competition  process — performance  work 
statement  (PWS)  and  most  efficient  organization  (MOE)  develop¬ 
ment,  independent  review,  and  contracting — are  highlighted  in  a 
separate  section.  However,  the  detailed  analysis  of  each  step  is  pro¬ 
vided  as  a  appendix.  Finally,  we  provide  our  conclusions  and 
recommendations.  Our  analysis  of  the  management  of  the  A-76  com¬ 
petition  process  will  be  issued  as  a  separate  report. 

There  are  also  several  other  appendices.  They  provide  our  methodol¬ 
ogy  for  assessing  the  expected,  observed,  and  effective  costs  of  the  16 
competitions  and  the  approach  we  used  to  compare  data  from  the  16 
competitions  with  CAMIS  data  for  completed  DoD  competitions. 
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Introduction 


President  Bush’s  Management  Agenda  identifies  14  areas  for  improv¬ 
ing  the  management  and  performance  of  the  federal  government. 
One  of  these  areas  is  competitive  sourcing,  or  the  public-private  com¬ 
petition  of  government  work  that  is  commercial  in  nature.  In  support 
of  the  President’s  Management  Agenda,  the  Office  of  Management 
and  Budget  (OMB)  has  set  targets  regarding  the  number  of  authori¬ 
zations  engaged  in  commercial  work  that  federal  agencies  must 
compete. 

Within  the  Department  of  Defense  (DoD),  large  multi-function  com- 

Q 

petitions'  are  a  popular  competitive  sourcing  approach  because  it  is 
believed  that  this  approach  generates  larger  savings  and  improved 
service  quality,  and  is  less  expensive  to  implement.  Moreover,  it  is  a 
way  to  meet  the  OMB  targets  with  fewer  competitions.  However,  there 
is  also  the  perception  that  it  limits  small  business  opportunities,  takes 
an  excessive  amount  of  time, 

To  understand  the  role  of  large-scale,  multi-function  competitions  in 
DoD’s  strategy  to  meet  the  President’s  Management  Agenda,  DoD 
asked  CNA  to  evaluate  16  competitions  in  this  category.  We  were  to 
identify  why  the  services  and  Defense  agencies  decided  to  conduct 
large  competitions,  how  they  conducted  the  competitions,  and 
whether  these  strategies  were  successful.  We  were  also  asked  to  deter¬ 
mine  the  savings  that  are  expected  and  the  costs  of  conducting  the 
competitions,  and  to  identify  economies  of  scale,  the  small  business 
opportunities,  the  implementation  times,  and  the  ease  of 


3.  These  large-scale  competitions  are  frequently  referred  to  as  BOS  com¬ 
petitions  because  they  commonly  include  many  of  the  support  func¬ 
tions  at  an  installation  or  base.  Because  there  is  no  uniform  definition 
of  what  constitutes  a  BOS  competition,  we  examined  large  competitions 
(100  FTEs  or  more  at  announcement)  that  included  one  or  more 
commercial  functions  or  activities. 
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competition.  As  a  point  of  comparison,  we  were  also  asked  to  examine 
several  multi-site,  single-function  competitions. 

The  16  competitions  were  completed  between  1996  and  2000,  and 
each  had  100  or  more  full-time  equivalents  (FTEs)  at  announcement. 
The  competitions  represent  $323  million  in  annual  pre-competition 
operating  costs  and  more  than  6,000  military  and  civilian  positions. 
They  were  conducted  by  the  Army,  Navy,  Air  Force,  and  the  Defense 
Logistics  Agency.  Thirteen  of  the  competitions  were  multi-function 
with  three  multi-site  competitions  used  for  comparison.  Our  analysis 
is  focused  on  the  competitive  sourcing  process  and  the  initial  imple¬ 
mentation  after  the  decision  to  either  contract  out  or  retain  the  work 
in-house.  This  report  documents  the  results  of  our  analysis. 


Background 


This  study  is  an  outgrowth  of  an  analysis  completed  by  CNAC  in  2001 
titled  The  Long-run  Costs  and  Performance  Effects  of  Competitive  Sourcing 
and  recent  data  collected  in  DoD  on  large  multifunction  competi¬ 
tions.  In  the  CNAC  study,  two  BOS  competitions  included  in  the  anal¬ 
ysis  showed  greater  than  expected  savings  after  competition.  Analysis 
of  the  DoD  data  indicate  that  large  multi-function  competitions 
decrease  the  time  and  cost  of  competition  by  combining  several  func¬ 
tions  into  a  single  competition,  thereby  reducing  redundancies  in 
competition  steps.  For  example,  with  a  large  single  competition  con¬ 
sisting  of  ten  functions,  the  steps  are  completed  once  instead  of  con¬ 
ducting  ten  separate  competitions  and  performing  the  steps  multiple 
times  over.  Large-scale  competitions  might  also  improve  service  qual¬ 
ity  by  reducing  the  coordination  necessary  with  multiple  service  pro¬ 
viders.  Based  on  these  two  examples,  the  analysis  of  the  savings 
associated  with  large-scale  competitions  and  an  assessment  of  time  to 
complete  was  warranted. 

If  greater  savings  can  be  achieved  from  large-scale  competitions,  DoD 
could  satisfy  the  President’s  mandate  with  fewer  competitions  while 
generating  greater  savings  for  use  in  meeting  mission  requirements 
and  improving  service  to  customers.  However,  there  are  concerns 
that  large-scale  competitions  may  limit  small  business  opportunities. 
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It  is  important,  therefore,  to  determine  whether  large-scale  competi¬ 
tions  offer  greater  advantages  than  smaller  competitions. 


Purpose 


The  purpose  of  this  study  is  to  assess  the  degree  to  which  large-scale 
competitions,  completed  since  1995,  have  resulted  in  increased  sav¬ 
ings  and  service  quality  and  decreased  the  costs  and  difficulty  of  con¬ 
ducting  the  competitions.  To  accomplish  this,  we  will  answer  the 
following  questions: 

•  Are  the  savings  from  large-scale  competitions  greater  than 
those  small  competitions? 

•  Does  service  performance  improve  with  large  competitions? 

•  Could  large  competitions  be  completed  in  less  time  than  if  the 
functions  were  competed  as  single  functions? 

•  Do  large  competitions  limit  small  business  opportunities? 

•  How  well  do  these  processes  work? 

Summary  of  competitions 

To  answer  these  questions,  we  examined  16  competitions  completed 
between  1996  and  2000.  We  chose  this  time  period  because  the  com¬ 
petitions  conducted  during  this  time  reflect  the  current  A-76  require¬ 
ments,  are  representative  of  functions  in  DoD’s  commercial  activities 
inventory,  and  would  likely  be  better  documented  than  earlier  com¬ 
petitions.  Table  1  lists  the  competitions  and  provides  summary  data 
on  their  service  affiliation,  size,  dollar  value,  and  resulting  service  pro¬ 
vider. 

The  16  competitions  consisted  of  12  contract  and  4  in-house  wins. 
They  represented  roughly  6,000  positions  (4,601  civilian  FTEs  and 
1,525  military  billets),  or  17  percent  of  the  FTEs/billets  competed 
between  1996  and  2000.  Annual  pre-competition  costs  for  these  com¬ 
petitions  totaled  about  $350  million.  The  competitions  ranged  in  size 
from  about  69  to  2000  FTEs,4  with  a  median  size  of  171  positions. 


4.  Announced  FTEs  were  100  or  greater.  Post-announcement  changes  to 
the  PWS  decreased  the  number  of  impacted  employes  for  this 
competition.  ^  ^ 


Table  1 .  Competition  summary  data 


Competition 

Service 

Annual  baseline 

Military  billets 

Civilian 

Result 

Type3 

cost 

FTEs 

BOS — transportation,  supply,  & 
material  maint. 

Army 

$13,460,313 

6 

254 

Contract 

MF/SS 

BOS — all  major  functions  (1 ) 

Navy 

$132,405,406 

201 

1,917 

Contract 

MF/SS 

Range  operations 

Navy 

$11,175,128 

0 

147 

In-house 

MF/SS 

Target  operations 

Navy 

$13,647,901 

0 

160 

Contract 

SF/MS 

BOS — all  major  functions  (2) 

Air  Force 

$48,910,684 

796 

238 

Contract 

MF/SS 

BOS — supply  &  transportation 

Air  Force 

$21,546,870 

252 

65 

Contract 

MF/SS 

Supply  (1) 

Navy 

$8,478,038 

0 

163 

Contract 

MF/SS 

Supply  (2) 

Air  Force 

$28,940,363 

0 

619 

Contract 

MF/SS 

Facilities  operation  and  mainte¬ 

Army 

$5,164,176 

0 

69 

In-house 

MF/SS 

nance 

Information  management 

Army 

$7,632,321 

0 

145 

In-house 

MF/SS 

BOS — all  major  functions  (3) 

Air  Force 

$9,645,797 

0 

163 

Contract 

MF/SS 

BOS — all  major  functions  (4) 

Air  Force 

$9,438,976 

0 

179 

Contract 

MF/SS 

Communications 

Air  Force 

$11,517,424 

190 

21 

Contract 

MF/SS 

Property  disposal 

DLA 

$4,794,521 

0 

100 

Contract 

SF/MS 

Technical  training 

Air  Force 

$9,603,061 

78 

79 

In-house 

SF/MS 

BOS — all  major  functions  (5) 

Army 

$13,719,851 

2 

282 

Contract 

MF/SS 

Total: 

$350,080,829 

1,525 

4,601 

a.  MF=  Multifunction,  SF=  Single  function,  MS=  Multiple  sites,  SS=  Single  site. 


Figure  3  illustrates  the  size  distribution  of  the  16  competitions.  The 
average  size  was  383  FTEs;  however,  this  average  is  skewed  by  a  single 
competition  with  more  than  2,000  positions. 

Distribution  by  service  and  type  of  competition 

Figure  4  shows  the  service  distribution  of  the  competitions  by  fre¬ 
quency  and  pre-competition  dollars.  The  Navy's  three  competitions 
represented  the  largest  percentage  of  pre-competition  dollars,  $157.2 
million  (45  percent) .  The  Air  Force  had  the  most  competitions  of  any 
single  service  or  agency  in  the  study  (six  competitions),  which 
together  accounted  for  32  percent  ($110.7  million)  of  the  pre-com¬ 
petition  dollars.  The  DLA's  three  competitions  represented  12  per¬ 
cent  ($42.2  million)  of  pre-competition  dollars,  and  the  Army's  four 
competitions  made  up  the  remaining  11  percent  ($40  million). 
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Figure  3.  Distribution  by  size  of  competition 


Competition 

BOS-all  major  functions  1 
BOS-all  major  functions  2 
Supply  2 

BOS-supply  &  transportation 
BOS-all  major  functions  5 
BOS-transpt.,  supply,  &  material  maint. 

Communications 
BOS-all  major  functions  4 
BOS-all  major  functions  3 
Supply  1 
Target  operations 
Technical  training 
Range  operations 
Information  management 
Property  disposal 
Facilities  operation  and  maintenance 


2,118 


1,034 


619 


317 

284 


260 


211 

179 

163 

163 

160 

157 

147 

145 


100 
]]  69 


500 


1,000  1,500 

Total  Positions  and  FTEs 


2,000 


Figure  5  shows  that  13  of  the  competitions  we  examined  were  multi¬ 
function/single  site  with  the  remaining  three  being  multi-site/single 
function  competitions. 
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Figure  4.  Service  distribution  of  the  1  6  competitions 


DLA  $42. 2M 

(12%) 


Navy 


Army 

4 


Figure  5.  Distribution  of  the  type  of  competition 
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Savings  from  large,  multi-function,  multi-site 
competitions 


This  section  provides  the  detailed  results  of  our  examination  of  the 
savings  rates  for  the  16  competitions.  Our  analysis  evaluates  and  com¬ 
pares  three  types  of  savings — expected,  observed,  and  effective — for 
each  competition.  In  addition,  we  analyzed  CAMIS  data  for  both 
single  and  multi-function  competitions.  Using  these  calculations,  we 
attempt  to  answer  the  following  questions: 

•  Did  the  16  competitions  we  reviewed  save  money? 

•  Are  the  savings  rates  from  large-scale,  multi-function  competi¬ 
tions  greater  than  those  from  small,  single-function 
competitions? 

•  How  do  the  number  of  functions  competed  and  the  size  of  the 
competition  affect  potential  savings? 

Evaluating  savings  of  the  16  competitions 

The  competitive  sourcing  program  was  initiated  in  order  to  reduce 
the  cost  of  needed  commercial  services  without  sacrificing  perfor¬ 
mance.  Savings  from  the  program  are  used  to  help  fund  moderniza¬ 
tion  and  improve  readiness  within  DoD.  Our  prior  work  examining 
the  program’s  cost  savings  as  well  as  similar  work  of  others  indicates 
that  overall  the  program  saves  30  percent  or  more  of  pre-competition 
costs.  This  does  not  mean  that  every  competition  results  in  significant 
cost  savings,  but  the  hallmark  of  a  successful  competition  is  reduced 
cost  of  operations. 

In  this  report,  we  use  the  same  methodology  we  used  in  our  report 
titled  “Long-Run  Cost  and  Performance  Effects  of  Competitive  Sourc¬ 
ing”  to  evaluate  costs  and  savings  from  competition.  We  begin  by 
reviewing  all  available  documentation  on  cost  and  performance; 
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interviewing  base  personnel  representing  management,  contracting 
and  customers;  and  examining  supplementary  data  such  as  audit 
reports  and  relevant  workload  data.  From  these  data,  we  identify  both 
the  pre-  and  post-competition  costs  of  providing  the  function  or 
activity. 

Once  pre-  and  post-competition  costs  are  isolated,  we  analyze 
changes  in  post-competition  cost  over  time.  Over  the  competition 
period,  there  are  revisions  to  the  original  performance  work  state¬ 
ment  (PWS)  that  reflect  changes  in  the  work  environment.  These 
changes  include  wage  increases  and  fluctuations  in  workload,  as  well 
as  changes  in  the  type  and  scope  of  work  to  be  performed.  For  exam¬ 
ple,  in  one  traditional  BOS  competition,  a  modification  of  $121,000 
was  approved  to  increase  the  vehicle  operations  and  maintenance 
workload  in  support  of  Operation  Noble  Eagle.  In  a  supply  and  trans¬ 
portation  competition,  the  government  identified  a  new  requirement 
for  container  handler  support. 

In  most  service  contracts,  Department  of  Labor  (DOL)  and/ or  Ser¬ 
vice  Contract  Act  (SCA)  prescribed  wage  rates  can  increase  total  con¬ 
tract  costs.  These  types  of  changes  affect  the  cost  of  providing  a 
particular  function  beyond  what  was  originally  identified  in  the  PWS. 
Many,  and  in  some  cases  most,  of  these  changes  occur  regardless  of 
whether  the  decision  is  to  contract  out  or  to  retain  the  function  in- 
house.  By  tracking  costs  over  time  and  isolating  changes  to  the  PWS, 
we  are  able  to  analyze  the  true — or  effective — costs  of  performing  the 
same  set  of  functions  originally  identified  in  the  PWS. 

In  our  evaluation  of  the  16  competitions  included  in  this  study,  we 
have  evaluated  and  compared  costs  and  savings  from  three 
perspectives:  expected,  observed,  and  effective.  Using  this  approach  allows 
us  to  separate  and  evaluate  the  costs  of  meeting  the  tasks  described  in 
the  original  PWS,  as  well  as  the  impact  on  costs  from  changes  in 
scope,  workload,  and  other  adjustments.  This  approach  also  allows  us 
to  assess  whether  long-run  savings  are  real  and  sustained  over  time. 
Definitions  of  terms  follow: 

•  Expected  costs  are  defined  as  what  the  government  expects  to  pay 
for  the  provision  of  a  commercial  function  after  a  competition 
is  completed  (e.g.,  the  price  of  the  winning  bid  plus  all  admin- 
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istrative  costs  to  the  government) .  Expected  savings  are  estimates 
of  the  difference  between  what  the  government  expects  to  pay 
and  the  pre-competition  costs  of  providing  the  function. 
Expected  costs  and  savings  are  forecasts  based  on  the  winning 
contract  or  most  efficient  organization  (MEO)  bid  at  the  time 
of  competition  and  can  be  incorporated  into  out-year  budget 
decisions. 

•  Observed  costs  are  defined  as  what  was  actually  spent  by  DoD  for 
the  provision  of  services.  Observed  costs  include  increases  or 
decreases  to  annual  costs  from  changes  in  scope,  workload, 
wages,  and  one-time  cost  adjustments.  Observed  savings  are  the 
difference  between  the  pre-competition  annual  costs  to  the 
government  and  the  actual  or  observed  costs  of  that  activity 
after  the  competition  was  completed. 

•  Effective  costs  are  defined  as  the  estimated  cost  to  DoD  of  provid¬ 
ing  the  same  set  of  services  as  originally  identified  in  the  cost 
comparison.  Effective  cost  estimates  exclude  cost  changes  that 
would  have  occurred  whether  or  not  the  activity  was  competed. 
In  the  BOS  competition  affected  by  Operation  Noble  Eagle, 
the  observed  costs  of  providing  services  increased  by  $121,000  in 
2001.  This  increase  was  due  to  additional  workload  needed  to 
support  our  military.  This  type  of  increase  in  workload,  and 
therefore  cost,  would  have  occurred  whether  the  necessary  ser¬ 
vices  were  provided  by  the  contractor  or  by  in-house  labor. 
Therefore,  the  effective  costs  for  2001  would  be  adjusted  to 
remove  these  one-time  costs.  By  adjusting  the  data  to  exclude 
workload,  scope,  wage,  and  one-time  costs,  effective  cost  esti¬ 
mates  allow  us  to  compare  changes  in  cost  while  keeping  the 
original  scope  constant.  Effective  savings  are  defined  as  the  dif¬ 
ference  between  the  pre-competition  annual  cost  to  the 
government  and  the  effective  costs  of  that  function  after 
adjustments  are  made.  Comparing  effective  and  pre-competition 
costs  provides  insight  into  true  cost  growth  or  savings. 

Effective  costs  are  the  most  meaningful  indication  of  whether  an  A-76 
competition  was  successful  in  producing  real  and  sustained  savings 
because  they  identify  the  costs  of  providing  the  same  scope  of  work 
over  time.  Measuring  effective  costs  and  savings  is  the  most  relevant 
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concept  for  policy  makers  to  use  in  assessing  the  value  of  the 
competitive  sourcing  program  and  identifying  any  needed  adjust¬ 
ments.  However,  it  is  also  important  to  examine  changes  in  observed 
costs  because,  historically,  these  are  the  types  of  costs  people  have 
looked  at  when  examining  the  worth  of  the  competitive  sourcing  pro¬ 
gram.  Installations,  in  particular,  focus  more  on  observed  costs  (as 
compared  to  effective  costs)  because  their  budgets  are  typically  based 
on  the  expected  costs  of  the  contract  or  MEO.  In  many  cases,  changes 
in  scope  and  workload  will  significantly  increase  the  observed  costs 
with  little  or  no  budgetary  relief.  A  detailed  description  of  this  meth¬ 
odology  is  included  in  appendix  C. 

After  our  evaluation  of  the  16  competitions,  we  present  a  comparison 
of  the  expected  savings  from  BOS  competitions  with  expected  savings 
from  single-function  competitions.  The  purpose  of  this  analysis  is  to 
determine  whether  BOS  competitions  result  in  greater  expected  sav¬ 
ings  than  the  average  DoD  A-76  competition. 

Limitations  of  analysis  of  cost 

As  mentioned  previously,  our  sample  includes  four  competitions 
which  resulted  in  the  function  being  retained  in-house.  The  current 
requirements  for  tracking  post-competition  data  on  personnel  and 
cost  do  not  provide  enough  detail  to  be  included  in  our  analysis  of 
observed  and  effective  cost.  It  is  difficult,  using  current  systems,  to 
track  the  impact  of  changes  in  scope  and  workload  to  changes  in  on¬ 
board  FTEs.  There  is  a  significant  lag  between  a  change  in  require¬ 
ments  and  the  date  at  which  positions  are  filled  or  vacated.  This  lag 
time,  coupled  with  normal  attrition,  clouds  the  ability  to  attach 
specific  costs  to  specific  changes  in  scope  or  workload. 

Further,  for  activities  where  the  pre-competition  staffing  was  heavily 
military,  delays  in  the  federal  hiring  process  can  cause  the  transition 
to  an  all-civilian  MEO  to  take  longer  than  expected.  These  lags 
during  transition  can  skew  both  observed  and  effective  costs,  making 
the  costs  of  the  in-house  organization  look  artificially  low.  Therefore, 
although  these  “in-house”  competitions  are  included  in  our  analysis 
of  expected  costs,  they  are  excluded  from  our  analysis  of  observed 
and  effective  costs. 
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In  selecting  the  competitions  for  our  study,  we  limited  our  sample  to 
those  studies  that  had  been  performed  between  1996  and  2000 
because  the  competitions  conducted  during  this  time  reflect  the  cur¬ 
rent  A-76  requirements;  are  representative  of  functions  in  DoD’s 
commercial  activities  inventory;  and  would  likely  be  better  docu¬ 
mented  than  earlier  competitions.  This  means,  however,  that  of  the 
16  competitions  we  studied,  7  competitions  (or  44  percent)  had  only 
one  year’s  worth  of  cost  data.  Although  this  is  sufficient  for  determin¬ 
ing  whether  the  PWS  was  written  comprehensively,  in  specific  exam¬ 
ples  it  can  be  insufficient  in  evaluating  the  full  cost.  For  example, 
significant  changes  can  take  place  after  the  first  year  and  be  applied 
retroactively.  Analysis  based  on  just  the  first  year’s  costs  can,  there¬ 
fore,  misrepresent  the  full  cost  of  operation.  Therefore,  it  is  impor¬ 
tant  to  interpret  the  results  carefully,  because  the  savings  realized  to 
date  for  the  16  competitions  can  change  by  the  end  of  the 
performance  period. 

Finally,  one  multi-site  competition  (property  disposal)  was  excluded 
from  our  analysis  of  effective  cost.  Information  on  specific  modifica¬ 
tions  to  the  contract  could  not  be  gathered  centrally.  Therefore,  only 
observed  and  expected  costs  are  included  for  this  competition. 

Analysis  of  the  16  competitions 

In  the  following  section,  we  present  the  results  of  our  cost  analysis  for 
the  16  competitions.  In  total,  the  estimated  annual  baseline  cost  for 
the  competitions  was  $350  million  with  expected  savings  of  48  per¬ 
cent  or  $168  million.  Expected  savings  ranged  from  12  to  75  percent, 
with  seven  of  the  competitions  having  an  expected  savings  rate  of  50 
percent  or  greater  (see  figure  6,  below) .  If  we  remove  the  largest  com¬ 
petition  from  this  group  (BOS-all  major  functions  (1)),  average 
annual  baseline  costs  remain  over  $217  million  and  expected  savings 
total  46  percent.  In  comparison,  the  population  of  competitions — 
both  small  and  large — in  the  Department  of  Defense  Commercial 
Activities  Management  Information  System  (CAMIS)  show  average 
expected  savings  of  33  percent. 

As  mentioned  above,  our  analysis  of  expected,  observed,  and  effective 
savings  rates  is  based  on  results  of  11  competitions  resulting  in  a 
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contract  win  (in-house  and  property  disposal  competitions  are 
removed  from  the  calculation).  When  we  limit  our  analysis  to  these 
1 1  competitions,  the  expected  savings  rate  increases  to  49  percent. 


Figure  6.  Distribution  of  expected  savings 


Percent  Savings 


In  examining  actual  or  observed  costs  for  the  1 1  contract  wins,  we 
found  that  observed  savings  ranged  from  6  to  74  percent,  with  an 
average  annual  observed  savings  of  44  percent.  Observed  costs  tend 
to  be  higher  than  expected  cost  because  they  include  increased 
expenditures  for  wage  increases,  scope  and  workload  changes,  and 
other  one-time  cost  adjustments.  The  observed  savings  rate  of  44  per¬ 
cent  can  be  viewed  as  the  most  conservative  savings  rate,  reflecting  all 
actual  costs  incurred  by  the  government  during  the  contract  period, 
regardless  of  whether  the  original  scope  of  work  has  increased.Table 
2  presents  the  expected,  observed,  and  effective  savings  rates  for 
thel6  competitions  we  examined. 

Although  observed  costs  reflect  the  full  outlay  of  funds  associated 
with  a  particular  contract,  effective  costs  reflect  the  true  costs  of  the 
providing  the  original  set  of  functions  outlined  in  the  PWS.  Changes 
to  scope  and  workload,  which  would  have  occurred  regardless  of 
whether  the  function  was  provided  via  an  in-house  or  contract  work¬ 
force,  are  removed  from  the  cost  analysis.  Our  analysis  found  that  the 
effective  savings  for  the  group  of  1 1  (the  contract  wins  minus  the  DLA 
property  disposal  competition)  ranged  from  11  to  75  percent  with  an 
average  of  49  percent  (see  figure  7) .  Weighting  the  numbers  to  deter¬ 
mine  the  savings  per  dollar  competed  yields  an  effective  savings  rate 
of  58  percent  (see  figure  8)  for  the  11  competitions  resulting  in 
contractor  provision  of  services. 


Table  2.  Savings  rates  for  the  1 6  competitions 


Function 

Service 

Average  annual 

Expected 

Observed 

Effective 

baseline  cost 

savings 

savings 

savings 

BOS — transportation,  supply,  &  material 
maint. 

Army 

$13,460,313 

21% 

15% 

19% 

BOS — all  major  functions  (1 ) 

Navy 

$132,405,406 

70% 

70% 

70% 

Range  operations 

Navy 

$11,175,128 

68% 

Target  operations 

Navy 

$13,647,901 

38% 

32% 

36% 

BOS — all  major  functions  (2) 

Air  Force 

$48,910,684 

47% 

41% 

45% 

BOS — supply  &  transportation 

Air  Force 

$21,546,870 

75% 

74% 

75% 

Supply  (1) 

DLA 

$8,478,038 

67% 

60% 

67% 

Supply  (2) 

DLA 

$28,940,363 

68% 

67% 

67% 

Facilities  operation  and  maintenance 

Army 

$5,164,176 

30% 

Information  management 

Army 

$7,632,321 

26% 

BOS — all  major  functions  (3) 

Air  Force 

$9,645,797 

12% 

6% 

11% 

BOS — all  major  functions  (4) 

Air  Force 

$9,438,976 

31% 

27% 

33% 

Communications 

Air  Force 

$11,517,424 

50% 

37% 

50% 

Property  disposal 

DLA 

$4,794,521 

33% 

43% 

Technical  training 

Air  Force 

$9,603,061 

66% 

BOS — all  major  functions  (5) 

Army 

$13,719,851 

65% 

59% 

65% 

Total: 

$350,080,829 

48% 

44% 

49% 

Grouping  the  1 1  competitions  by  range  of  savings,  we  found  that  the 
largest  number  of  competitions  had  effective  savings  of  50  percent  or 
greater.  In  addition,  in  terms  of  the  size  of  the  competition,  the  larg¬ 
est  competitions  also  generated  the  greatest  savings.  For  those  com¬ 
petitions  that  saved  50  percent  or  more  of  pre-competition  costs,  the 
average  annual  baseline  cost  was  over  $36  million,  versus  roughly 
$11.5  million  for  those  competitions  that  saved  30  percent  or  less. 
Figure  9  summarizes  the  distribution  of  effective  savings  by  number 
of  competitions,  and  figure  10  groups  them  by  dollar  value. 

In  our  evaluation  of  expected  savings,  the  results  are  the  same — that 
is,  the  larger  competitions  in  our  sample  showed  greater  “expected” 
savings.  This  finding  is  consistent  with  our  past  research  which  has 
found  that  expected  savings  increase  with  the  size  of  the  competition. 
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Figure  7.  Savings  rates  for  1 1  competitions 
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Figure  8.  Savings  rates  for  1 1  competitions  (weighted) 
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Figure  9.  Distribution  of  effective  savings  by  number  of 
competitions  (1 1  competitions) 
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Figure  10.  Distribution  of  effective  savings  by  dollar 
value  (1 1  competitions) 


Percent  Savings 


Comparison  to  the  population  of  DoD  competitions 

To  determine  whether  the  48-percent  average  expected  savings  for 
the  16  competitions  was  consistent  with  the  expected  savings  for  all 
multiple-function  competitions,  we  also  examined  the  expected  sav¬ 
ings  rates  for  all  multiple-function  competitions  conducted  by  DoD 
between  1978  and  January  2001,  as  recorded  in  CAMIS.5  We  then 
compared  their  savings  rates  with  those  of  single-function  competi¬ 
tions. 

The  question  facing  competitive  sourcing  decision-makers  is  whether 
it  is  more  beneficial  to  perform  multiple  small,  single-function  com¬ 
petitions  or  combine  them  into  one  large  multiple-function 
competition.  The  results  of  our  analysis  indicate  that  a  positive 


5.  CAMIS  is  the  Commercial  Activities  Management  Information  System. 
It  records,  among  other  data,  the  expected  savings  for  every  competi¬ 
tion  DoD  has  completed  since  the  inception  of  the  competitive  sourc¬ 
ing  program.  There  were  1,729  competitions  of  installation  services, 
other  non-manufacturing  operations,  and  maintenance  of  real  property 
that  were  conducted  between  1978  and  2001.  These  competitions  rep¬ 
resent  about  74  percent  of  all  competitions  conducted  during  that  time 
frame. 
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relationship  exists  between  competition  size  and  potential  competition 
savings,  yet  a  negative  relationship  exists  between  the  number  of  func¬ 
tions  in  a  competition  and  expected  savings.  The  benefits  associated  with 
increasing  competition  size,  however,  outweigh  the  costs  from  including 
additional  functions.  Therefore,  in  a  real-world  setting  where  the  pri¬ 
mary  method  for  increasing  competition  size  is  to  combine  several  func¬ 
tions  into  a  large,  multiple-function  study,  where  practicable,  decision¬ 
makers  should  use  this  approach. 

Multi-function  versus  single-function 

The  DoD  Competitive  Sourcing  Program  does  not  have  a  uniform  defi¬ 
nition  of  what  constitutes  a  multiple-function  study.6 7 8  In  this  analysis,  we 
relied  on  several  variables  to  identify  the  effects  of  packaging  competi¬ 
tions  into  single-function  competitions  or  multi-function  ones.  We  used 
a  series  of  regression  models  that  include  these  variables  to  measure  the 
effects  of  combining  functions  and  competing  activities  across  multiple 
sites.  We  categorized  studies  into  one  of  three  categories:  1)  competitions 
with  only  one  function  code,  2)  competitions  with  multiple  function 
codes,  or  3)  a  whole  base  competition — designated  with  a  function  code 
of  PI 00  in  the  CAMIS  files.  We  grouped  studies  in  this  manner  to  exam¬ 
ine  the  effects  of  combining  competitions  without  making  a  subjective 
determination  as  to  whether  a  competition  was  indeed  a  single-  or  multi- 

.•78 

pie-function  competition.  Table  3  provides  some  basic  characteristics 
for  each  of  these  three  categories. 


6.  A  recent  change  to  the  required  fields  in  CAMIS  requires  a  study  to  be  iden¬ 
tified  as  either  a  single-function  or  multiple-function  competition.  However, 
a  standard  definition  still  has  not  been  established,  leaving  it  up  to  the 
claimants  or  the  individual  installations  to  determine  the  characteristics  of 
a  multiple  function  competition.  Because  multiple-function  studies  are 
allowed  more  time  to  be  completed,  the  incentive  is  to  classify  a  competition 
as  being  a  multiple-function  competition  whenever  possible. 

7.  Determining,  for  example,  whether  several  function  codes  that  describe 
what  is  in  effect  a  only  one  activity,  such  as  motor  vehicle  operations  and 
maintenance,  should  be  classified  as  a  single-  or  multiple-function 
competition. 

8.  Designating  a  competition  as  being  either  a  single-  or  multiple-function 
competition  determines  the  amount  of  time  allowed  for  completion  under 
OMB  circular  A-76.  Under  current  guidelines,  single-function  studies  are 
allowed  24  months  while  multiple-function  competitions  are  given  48 
months  to  reach  final  decision. 


Table  3.  Competition  characteristics  by  competition  type 


Percentage 


Type3 

Competitions. 

Mean 

Median 

savings 

Single 

1,436 

26 

14 

32% 

Multiple 

321 

85 

40 

34% 

Multiple-P 

35 

230 

130 

32% 

Total 

1,792 

40 

16 

32% 

a.  Single — one  function  code;  Multiple — more  than  one  function  code;  Multiple — 
P-Function  code  PI  00. 


Most  of  the  studies — about  80  percent — involved  competing  only  one 
function  code.  Competitions  with  multiple  function  codes  make  up 
about  18  percent  of  the  sample  whereas  only  about  2  percent  of  the 
studies  were  designated  by  function  code  P100.  The  estimated  per¬ 
centage  savings9  increases  slightly  for  multiple  function  studies,  but 
this  does  not  control  for  other  competition  characteristics  that  could 
affect  potential  savings.  In  other  words,  factors  such  as  the  size  of  the 
competition  or  the  composition  of  the  billets  included  could  be 
driving  this  observed  difference. 


Results 


The  number  of  positions  and  the  composition  of  the  activity  com¬ 
peted  affect  the  potential  savings  that  can  be  achieved.  The  size  of  a 
competition  can  be  increased  in  one  or  a  combination  of  three  ways. 
The  number  of  positions  included  in  a  competition  can  be  increased 
by  adding  additional  positions  that  perform  the  same  function  as 
those  being  competed  in  a  separate  part  of  the  base,10  adding  addi¬ 
tional  functions,  and  competing  the  activity  across  multiple  locations. 
The  motivation  for  increasing  the  size  of  a  competition  is  that  evi¬ 
dence  suggests  increasing  size  leads  to  additional  savings  from  poten¬ 
tial  economies  of  scope  or  scale.  However,  adding  additional 


9.  In  our  discussion  of  percentage  savings,  we  are  referring  to  “expected” 
savings  based  on  information  at  the  time  of  study  completion. 

10.  For  example,  including  all  billets  in  warehouse  operations  rather  than 
including  only  the  billets  in  the  warehouse  function  of  the  activity  being 
studied. 
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functions  and/  or  locations  increases  the  complexity  of  the  competi¬ 
tion.  We  would  expect,  holding  all  else  constant,  that  making  a  com¬ 
petition  more  complex  would  decrease  the  potential  savings.  The 
question  facing  decision-makers  for  DoD’s  commercial  activities  pro¬ 
gram  is  whether  it  is  better  to  compete  a  number  of  small  activities  or 
to  combine  them  into  one  large  competition. 

We  estimated  a  Tobit  regression  model  to  quantify  the  effects  of  pack¬ 
aging  and  size  on  overall  competition  savings.  The  relevant  portion  is 
reproduced  in  this  section;  a  description  of  the  model  and  a  complete 
listing  of  the  estimation  results  is  provided  in  appendix  B. 


Size 


Table  4  lists  the  regression  coefficients  for  the  variables  describing  the 
size  of  the  competition.  The  coefficients  are  statistically  significant 
and  indicate  that,  holding  all  else  constant,  increasing  the  size  of  a 
competition  does  increase  the  potential  savings.  This  reaffirms  previ¬ 
ous  findings  that  found  a  positive  relationship  between  competition 
size  and  potential  savings.  The  effect  of  increasing  size  on  competi¬ 
tion  savings  is  greater  for  smaller  competitions  than  it  is  for  larger 
competitions.  Figure  11  shows  the  predicted  competition  savings 
based  on  the  size  of  the  study. 


Table  4.  Regression  coefficients  for  size-of-competition  variables 

Variable  Coefficient  Standard  error  P-value 

LNBS  0.128  0.018  0.00 

LNBS2  -0.012  0.003  0.00 


The  predicted  savings  based  on  the  estimated  coefficients  shows  a  sig¬ 
nificant  slowing  in  the  additional  savings  of  each  position  added 
starting  at  about  150  positions. 

Impact  of  multi-function 

Increasing  the  number  of  functions  and/ or  sites  in  a  study  can  lead 
to  a  competition  covering  a  larger  number  of  billets — which  could 
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Figure  1 1 .  Expected  savings  and  size  of  study 
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increase  savings — but  it  also  increases  the  complexity  of  the  competi¬ 
tion — which  could  reduce  savings.  We  included  indicator  variables  in 
the  Tobit  regression  model  for  multiple-function  competitions  as 
defined  above.  The  relevant  portion  of  the  regression  model  is  repro¬ 
duced  in  table  5.  Both  coefficients  are  negative  and  statistically 
significant  which  indicates  that  both  reduce  the  potential  savings 
from  competition.  The  values  of  the  coefficients  indicate  that,  at  the 
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mean,  multiple-function  studies  and  studies  designated  as  GA  func¬ 
tion  code  P100  realized  about  2.7  percentage  points  and  4.5  percent¬ 
age  points  lower  savings  than  single-function  competitions, 
respectively.11  This  result  is  holding  all  other  variables  constant, 
including  size.  In  a  real-world  setting,  the  size  of  the  competition 
would  increase  with  the  number  of  functions  added.  Based  on  the 
results  in  the  previous  section,  the  size  of  a  competition  would  need 
to  increase  by  10  positions  and  25  positions  to  offset  the  negative 
effect  on  savings  from  combining  functions  in  multi-function  and 
PI 00  competitions,  respectively.  The  mean  value  of  competition  size 
is  17  positions.  However,  this  result  is  based  on  the  categories  used  in 
this  analysis;  caution  should  be  used  when  extrapolating  the  estimates 
beyond  these  definitions.  These  broad  categories  cannot  be  used  to 
decompose  the  effect  of  various  combinations  of  functions  on  poten¬ 
tial  competition  savings.  We  expect  that  economies  of  scale  or  scope 
would  be  less  likely  when  combining  functions  that  are  unrelated 
than  when  combining  functions  that  are  related.  For  example,  fewer 
economies  would  be  available  when  combining  intermediate  aircraft 
(function  codeJ501)  with  family  services  (function  code  G904)  than 
when  combining  it  with  intermediate  maintenance  of  aircraft  engines 
(function  codeJ502). 


Table  5.  Regression  coefficients  for  complexity-of-competition 
variables 


Variable 

Coefficient 

Standard  error 

P-value 

UNIQ_STS 

-0.008 

0.011 

0.46 

BOS2 

-0.033 

0.015 

0.03 

BOS3 

-0.054 

0.027 

0.04 

1 1 .  We  estimated  several  additional  regressions  in  an  attempt  to  isolate  the 
effects  of  each  additional  function  and  function  group  on  potential 
competition  savings,  but  the  results  were  similar  to  those  shown  in  table 
5. 
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Impact  of  multi-site 


The  coefficient  representing  the  number  of  different  sites  included 
in  each  study  is  statistically  insignificant.  This  variable  tracks  the 
number  of  distinct  UICs  included  in  the  study  but,  unfortunately,  we 
do  not  know  the  proximity  of  the  sites.  We  would  expect  a  site  located 
20  miles  away  to  face  additional  logistical  problems  that  sites  located 
on  the  other  side  of  the  installation  would  not.  The  lack  of  statistical 
significance  could  be  caused  by  the  low  number  of  competitions  held 
across  multiple  locations  and  the  correlation  between  this  variable 
and  the  definitions  of  the  multiple-function  categories.  Only  about  6 
percent  of  the  multiple-function  competitions  and  1  percent  of 
single-function  competitions  were  competed  across  multiple  sites. 
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Post-competition  performance  of  large  multi¬ 
function,  multi-site  competitions 


Sustained  quality  performance  is  critical  to  safeguard  DoD’s  assets 
and  satisfy  mission  requirements.  A  competition  cannot  be  deemed  a 
success  if  savings  are  achieved,  but  performance  suffers  commensu- 
rately.  Although  there  might  be  an  initial  drop  in  performance  as  the 
new  provider  becomes  familiar  with  the  performance  requirements, 
he  or  she  should  be  at  full  operational  performance  by  the  end  of  the 
agreed-upon  transition  period. 

Based  on  the  interviews  we  conducted,  customers,  managers,  and 
contracting  personnel  thought  that,  in  general,  the  outlook  for  per¬ 
formance  in  the  future  was  either  improving  or  steady.  In  only  one 
competition  did  they  believe  that  performance  was  likely  to  slip.  And 
in  one  competition,  both  the  future  requirements  and  the  relation¬ 
ship  between  the  contractor  and  the  installation  were  being  negoti¬ 
ated,  and  installation  officials  were  uncomfortable  specvdating  on 
performance.  Because  of  the  lack  of  data  in  this  case,  we  have 
excluded  it  from  the  overall  evaluation  of  performance.  However,  we 
have  included  a  separate  discussion  of  this  competition  in  appendix 
D.  Figure  12  summarizes  the  results  of  our  interviews. 

Overall  performance 

We  interviewed  management,  contracting  personnel,  and  customers 
on  their  perception  of  post-competition  performance.  They  were 
asked  a  series  of  questions  related  to  performance  of  the  tasks 
defined  in  the  PWS  and  asked  to  rank  their  perceptions  on  a  scale  of 
one  (dissatisfied  with  performance)  to  five  (very  satisfied  with  perfor¬ 
mance).  We  also  asked  them  to  rank  their  perceptions  of  perfor¬ 
mance  prior  to  competition  in  order  to  provide  a  point  of 
comparison. 
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Figure  12.  Outlook  for  the  quality  of  performance  in  the  future 


V  3 

c 

o 


CD 

Q_ 


E 

o 

U 


0) 

-Q 

E 

D 

z 


Improving  Steady  Slipping  Unknown 


In  nearly  all  of  these  cases,  the  personnel  we  interviewed  believed  that 
overall  performance  after  competition  was  neutral  (meeting  minimum 
standards  defined  in  the  PWS)  to  satisfactory.  For  this  group,  the  average 
satisfaction  level  was  3.6.  Table  6  illustrates  the  post-competition  perfor¬ 
mance  rating  by  competition  and  by  personnel  type. 

Figure  13.  Comparison  of  pre-  and  post-competition  performance  rating 
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Table  6.  Post-competition  performance  rating  by  competition  and  by 
personnel  type 


Goirpetitions 

Customer 

Contracting  Functional 

Officers  Managers  CVerall 

1  BC6— transportation,  supply,  &  material  maint. 

•  • 

2  BC6— all  major  functions  (1) 

• 

•  • 

3  Range  operations 

• 

• 

4  Target  operations 

• 

•  • 

5  BC6— all  major  functions  (2) 

• 

• 

9  • 

6  BC6— supply  &  transportation 

7  Supply  (1) 

• 

• 

•  • 

8  Supply  (2) 

• 

m 

• 

9  Facilities  operation  and  maintenance 

• 

•  • 

10  Information  management 

• 

•  • 

1 1  BC6— all  major  functions  (3) 

• 

• 

•  • 

12  BC6— all  major  functions  (4) 

• 

§ 

•  • 

13  Gorrmunications 

• 

• 

®  # 

14  Property  disposal 

• 

•  • 

15  Technical  training 

16  BC6— all  major  functions  (5) 

• 

• 

•  • 

CVerall  Average 

• 

• 

•  • 

0  (Blue)  Very  satisfied  with  performance  (4.0  -  5.0) 
(Green)  Neutral/satisfied  with  performance  (2.6-3.9' 
w  (Red)  Dissatisfied  with  performance  (1 .0-2.5) 


In  two  competitions,  the  personnel  we  interviewed  found  that  post¬ 
competition  was  generally  better  than  performance  prior  to  competi¬ 
tion.  In  five  competitions,  personnel  found  that  post-performance  was 
worse.  However,  four  of  these  competitions  were  in  their  first  or  second 
year  of  performance,  indicating  performance  problems  that  may  be 
transitional.  Three  of  the  four  competitions  retained  in-house  showed 
pre-  and  post-competition  performance  essentially  remaining 
unchanged.  Figure  13  provides  a  comparison  of  pre-  and  post-competi¬ 
tion  performance  ratings,  remaining  unchanged.  The  12  competitions 
with  neutral  to  satisfactory  ratings  accounted  for  80  percent  of  the  total 
dollar  value  of  the  competitions  in  this  analysis.  Three  competitions 
received  an  overall  satisfaction  rating  above  4.0,  indicating  that  person¬ 
nel  were  very  satisfied  with  the  provisions  of  services.  These  three  high¬ 
est-rated  competitions  accounted  for  12  percent  of  the  examined 
competitions’  total  dollar  value. 
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Competitions 


Figure  14  shows  the  distribution  of  the  competitions’  performance  by 
number  and  cost.  The  one  competition  where  interviewees  indicated 
they  were  dissatisfied  with  performance  was  a  supply-function  compe¬ 
tition.  It  accounted  for  $28.9  million  (8  percent)  of  the  $350  million 
in  total  pre-competition  costs  for  services  of  the  competitions  we 
examined. 


Figure  14.  Distribution  of  competition  performance  by  number  and  cost 
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In  the  cases  where  installation  personnel  were  dissatisfied  with  the 
level  of  performance,  the  reasons  they  gave  for  poor  performance 
included: 

•  The  new  provider  was  unable  to  hire  sufficient  numbers  of 
trained  personnel. 

•  Subcontractors  had  little  prior  experience  and  poor  manage¬ 
ment. 

•  Preventive  maintenance  was  not  a  high  enough  priority  with 
the  new  provider. 

•  Complaints  were  difficult  to  resolve  and  required  extensive 
debate  of  the  requirements  in  the  PWS. 
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Performance  over  time 


In  our  previous  research  on  competitive  sourcing,  we  found  that  tran¬ 
sition  issues  tended  to  affect  performance  during  the  first  year  but 
that  overall  performance  was  viewed  as  satisfactory  in  subsequent 
years.  Our  findings  on  the  performance  of  the  16  large,  multi-func¬ 
tion  or  multi-site  competitions  are  consistent  with  these  earlier  find¬ 
ings.  Seven  of  the  eight  competitions  where  personnel  rated 
performance  below  pre-competition  levels  were  in  their  first  or 
second  year  of  performance.  Further,  as  discussed  in  the  beginning 
of  this  section  on  overall  performance,  in  all  but  one  competition 
performance  is  improving  or  expected  to  improve.  These  findings  on 
performance  indicate  that  the  same  types  of  performance  challenges, 
during  transition,  affect  both  large-  and  small-scale  competitions. 

Performance  by  type  of  personnel  interviewed 

When  we  compared  our  interviewees’  view  of  performance  by  the 
type  of  personnel  interviewed,  we  found  that  the  customers  rated  per¬ 
formance  lower  than  did  contracting  personnel  or  functional  manag¬ 
ers.  This  is  opposite  to  our  findings  in  our  report  on  the  long-run 
costs  and  performance  of  A-76  competitions.  In  the  earlier  report,  we 
found  that  customers  generally  rated  performance  higher  than  did 
contracting  personnel  or  functional  managers.  Figure  15  illustrates 
our  findings. 

With  these  15  competitions,  we  found  that  customers  rated  post-com¬ 
petition  performance  slightly  below  pre-competition  performance 
and  with  lower  values  as  compared  to  contracting  personnel  and 
functional  managers.  Because  customers  generally  focus  on  the  time¬ 
liness  and  quality  of  service  that  they  receive,  some  of  this  differential 
may  be  explained  by  the  fact  that  some  transition  issues  had  not  been 
resolved  at  the  time  of  this  analysis. 

Although  some  of  the  differential  between  customers,  functional 
managers,  and  contracting  officers  may  be  explained  by  transition 
issues,  the  competition  with  the  greatest  difference  in  how  customers 
rated  performance,  however,  was  an  aircraft  maintenance  competi¬ 
tion  in  its  third  year  of  contract  performance.  In  this  example, 
contracting  and  functional  managers  were  extremely  satisfied  with 
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Figure  1 5.  Performance  by  type  of  personnel  interviewed 


Customer  Contracting  Functional 
Officers  Managers 


■  Very 
Satisfied 

□  Neutral/ 
Satisfied 

■  Dissatisfied 


the  contractor’s  performance.  The  customers,  who  were  military 
pilots,  expressed  a  strong  preference  for  the  pre-competition  perfor¬ 
mance  by  military  personnel.  We  also  observed  the  same  preference 
during  our  examination  of  the  technical  training  competition  and  an 
aircraft  maintenance  function  in  our  previous  report.  In  all  three 
competitions,  the  customers,  predominantly  military,  believed  that 
military  personnel  were  more  conscientious  than  either  civilian  or 
contract  employees. 

In  general,  both  contract  personnel — who  were  concerned  with  both 
performance  requirements  and  how  well  contractors  meet 
administrative  and  procedural  requirements — and  functional  man¬ 
agers  and  quality  assurance  personnel — who  are  knowledgeable 
about  performance  requirements  both  before  and  after  competi¬ 
tion — rated  post-competition  performance  as  equivalent  to  pre-com¬ 
petition  performance.  The  contract  personnel  rated  post¬ 
competition  performance  slightly  higher  than  pre-competition  per¬ 
formance,  whereas  functional  managers  ranked  it  in  reverse.  Interest¬ 
ingly,  their  perception  of  post-competition  performance  was  about 
the  same.  It  was  their  views  on  pre-competition  performance  that  was 
the  most  different.  Functional  managers  believed  that  pre-competi¬ 
tion  performance  was  higher  than  did  contracting  personnel. 
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Length  of  the  process 


19. 

DoD  is  required  by  law  to  complete  its  competitions  within  certain 
time  limits.  One  of  the  questions  we  were  asked  to  address  was 
whether  a  single  BOS  competition  can  be  completed  more  quickly 
than  if  the  component  functions  and/ or  sites  were  each  competed 
separately.  We  found  that  the  16  competitions: 

•  Experienced  bottlenecks  during  the  process  for  a  variety  of  rea¬ 
sons  including  poor  coordination  between  the  affected  parties, 
small  business  considerations,  and  acts  of  nature 

•  Took  the  same  or  less  time  to  compete  than  the  population  of 
A-76  competitions  DoD  completed  during  roughly  the  same 
time  period 

•  Were  well  within  the  statutory  time  limit  imposed  on  DoD  com¬ 
petition 

•  Showed  no  correlation  between  competition  size  and  comple¬ 
tion  time,  but  rather  seemed  to  be  primarily  affected  by  the 
degree  to  which  the  competition  manager  drove  the 
competition. 

Bottlenecks  to  the  process 

Delays  are  common  during  the  course  of  a  competition.  Although 
some  delays  can  be  anticipated  and  actions  taken  to  avoid  them, 
others  are  unavoidable.  Half  of  the  competitions  we  examined  expe¬ 
rienced  little  or  no  delay.  One  competition  experienced  a  4-month 
delay  because  of  a  severe  storm  of  hurricane  or  typhoon  proportions. 


12.  Statute  has  required  single-function  competitions  to  be  completed  within 
24  months  and  multi-function  competitions  to  be  completed  within  48 
months.  Recent  changes  in  A-76  will  shorten  time  lines  to  a  maximum  of  18 
months. 
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Four  competitions,  or  25  percent,  experienced  delays  of  6  months  or 
more.  Of  those  with  delays,  the  most  common  reason  was  problems 
in  coordinating  the  competition  between  the  headquarters  organiza¬ 
tions  and  installations. 

Some  of  the  coordination  problems  were  due  to  differing  expecta¬ 
tions  and/ or  incentives  between  the  various  groups.  For  example,  in 
one  competition,  there  were  disagreements  between  the  A-76  team 
and  the  functional  managers  on  the  level  of  specificity  and  degree  of 
prescription  to  use  in  the  PWS.  In  another  competition,  there  was  dif¬ 
ficulty  in  getting  one  of  the  two  installations  to  perform  its  assigned 
role.  In  a  third  competition,  multiple  headquarters  reviews  and  coor¬ 
dination  of  several  ongoing  competitions  delayed  the  progress  of 
individual  competitions. 

Four  competitions  were  delayed  while  the  respective  teams  negoti¬ 
ated  with  representatives  of  the  Small  Business  Administration  (SBA) . 
These  negotiations  primarily  addressed  whether  existing  small  busi¬ 
ness  contracts  would  be  included  in  the  competition’s  scope,  or  to 
identify  the  level  of  small  business  subcontracting  goals  that  would  be 
included  if  the  competition  resulted  in  a  contract  award.  Generally, 
the  competition  managers  agreed  to  exclude  existing  contracts  from 
the  scope  of  their  competitions.  In  one  competition,  the  contracting 
officer  wanted  to  issue  an  unrestricted  solicitation,  but  SBA  wanted  it 
restricted  to  section  8a  firms.  The  solicitation  was  finally  issued  as  a 
restricted  competition.  Installation  officials  said  that  the  decision  to 
restrict  the  solicitation  to  small  businesses  limited  the  number  of 
functions  they  could  include  in  the  PWS  and  the  number  of  firms  that 
would  have  bid.  They  said  that  the  negotiations  to  reach  the 
compromise  delayed  the  competition  by  4  or  5  months. 

16  competitions  versus  DoD  competitions 

To  address  the  question  of  whether  a  multi-function  competition  can 
be  completed  more  quickly  than  if  the  component  functions  and/ or 
sites  were  each  competed  separately,  we  compared  the  completion 
times  of  our  sample  of  16  competitions  with  the  population  of  DoD 
competitions  completed  during  the  same  time  frame,  1995  to  2001, 
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as  well  as  those  completed  from  1978  to  1994.  The  population  of 
competitions  included  both  single-  and  multi-function. 


Table  7.  Median  competition  com 

pletion  times3 

Competitions 

Number  completed 

Median  months  to  completion 

1 6  competitions 

16 

25 

Before  1 995 

2,043 

27 

After  1 995 

287 

26 

Total  1978-2001 

2,330 

27 

a.  Excludes  direct  conversion,  streamlined  cost  comparison,  in-progress  and 
canceled  competitions. 


As  we  see  in  table  7,  the  16  competitions  did  not  appear  to  take  any 
longer  than  the  other  competitions  completed  during  the  same  time 
period,  and  the  competitions  competed  prior  to  1995.  All  16  were 
completed  well  within  their  statutory  limit  of  48  months.  Half  the 
competitions  even  fell  within  the  24-month  period  required  for 
single-  function  competitions. 

We  also  analyzed  the  completion  times  of  other  multi-function  and 
single-function  competitions  with  much  the  same  results.  Multi-func¬ 
tion  competitions  took  an  average  of  27.9  months  to  complete, 
whereas  single-function  competitions  took  an  average  of  22.4 
months.  The  average  size  of  these  competitions  was  138  FTEs  and 
33.9  FTEs,  respectively. 

Correlation  between  completion  times  and  size  and  other  factors 

There  appears  to  be  no  correlation  between  the  size  of  the  16  BOS 
competitions  and  their  durations.  For  example,  the  two  largest  com¬ 
petitions,  each  with  more  than  1,000  FTEs,  took  about  29  months  to 
complete  whereas  one  of  the  smaller  competitions  with  about  170 
FTEs  took  38  months.  Nor  does  there  appear  to  be  any  correlation 
between  duration  and  whether  the  competition  is  multi-function  or 
multi-site.  Figure  16  presents  these  results. 
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Figure  1  6.  Size  (total  FTEs)  versus  length  of  competition 
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However,  the  completion  times  did  seem  to  be  affected  by  the  leader¬ 
ship  of  the  competition  manager  and  diligence  of  the  A-76  team.  For 
example,  the  competition  manager  for  the  largest  competition  we 
examined  actively  managed  the  competition.  A  centrally  managed 
competition,  it  was  originally  scheduled  for  completion  within  12 
months.  However,  a  typhoon,  lengthy  negotiations  with  SBA,  and  the 
need  to  significantly  revise  the  in-house  proposal  delayed  its  compe¬ 
tition.  Although  each  of  these  bottlenecks  caused  delays  of  4  to  6 
months,  the  competition  manager  and  the  work  of  those  involved 
were  able  to  keep  the  delays  as  short  as  possible  and  to  the  mitigate 
their  effect  on  the  overall  schedule.  They  kept  the  competition 
moving  so  that  the  competition  was  completed  in  28  months. 

Similarly,  a  smaller,  installation-managed  competition  was  beset  with 
disagreements  between  the  PWS  team,  the  contracting  organization, 
and  the  functional  managers.  In  addition,  the  initial  draft  PWS 
included  existing  contracts  within  its  scope  which  were  later 


removed,  requiring  significant  revisions  to  the  PWS.  The  representa¬ 
tives  from  the  functions  wanted  the  PWS  to  be  very  detailed  and  pre¬ 
scriptive,  and  their  initial  draft  was  not  reflective  of  the  work  being 
competed.  Representatives  from  the  contracting  organization,  on  the 
other  hand,  wanted  a  performance-based  document.  Similar  prob¬ 
lems  existed  within  the  MEO  team.  The  competition  manager  from 
an  independent  organization  resolved  these  problems  by  bringing  in 
technical  people  with  a  business  perspective  to  redraft  the  PWS  and 
removed  the  co-chair  of  the  MEO  team,  a  functional  representative. 
The  competition  was  completed  in  22  months,  4  months  behind  its 
original  18-month  schedule. 

In  contrast,  an  installation-managed  medium-sized  BOS  competition 
took  37  months  to  complete.  The  competition  encountered  prob¬ 
lems  such  as  conflicting  headquarters  guidance,  SBA  negotiations, 
independent  review  certification,  and  support  contractor  availability 
added  from  15  to  20  percent  additional  time.  For  example,  the  only 
support  contractor  with  A-76  experience  that  the  installation  consid¬ 
ered  was  not  on  the  headquarters  list  of  contractors  for  which  the 
headquarters  paid.  It  took  the  installation  additional  time  to  get 
headquarters’  agreement  to  fund  the  contractor.  In  addition,  several 
installation  officials  had  the  expectation  that  the  contractor  was  to 
“help  the  in-house  win”  and  were  disappointed  in  his  performance  as 
a  consequence.  The  largest  problem  with  this  type  of  management 
model  was  that  the  government  personnel  in  charge  of  the  process 
itself  had  little  to  no  experience  with  A-76  competitions.  At  each 
point  in  the  process  they  were  “re-inventing  the  wheel.”  Table  8  pre¬ 
sents  the  duration  and  management  type  for  the  16  competitions. 
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Table  8.  Duration  and  management  type  for  the  1 6  competitions 


Function 

Service/ 

agency 

Size  (FTE) 

Duration 

(months) 

Type  of  management 

BOS — transportation,  supply,  &  material 
maint. 

Army 

362 

37 

Installation 

BOS — all  major  functions  (1) 

Navy 

2,118 

28 

HQ 

Range  operations 

Navy 

147 

26 

Installation 

Target  operations 

Navy 

160 

38 

Installation 

BOS — all  major  functions  (2) 

Air  Force 

1,034 

28 

HQ  mgt/lnstallation 
perf. 

BOS — supply  &  transportation 

Air  Force 

317 

22 

HQ  mgt/lnstallation 
perf. 

Supply  (1) 

DLA 

166 

21 

HQ 

Supply  (2) 

DLA 

552 

23 

HQ 

Facilities  operation  and  maintenance 

Army 

340 

24 

Installation 

Information  management 

Army 

87 

23 

Installation 

BOS — all  major  functions  (3) 

Air  Force 

113 

27 

HQ 

BOS — all  major  functions  (4) 

Air  Force 

172 

38 

HQ 

Communications 

Air  Force 

211 

22 

Installation 

Property  disposal 

DLA 

100 

22 

HQ 

Technical  training 

Air  Force 

157 

22 

HQ  mgt/lnstallation 
perf. 

BOS — all  major  functions  (5) 

Army 

284 

27 

Installation 
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Advantages  and  disadvantages  of  multi- function 
competitions 

Results  of  interviews 

During  our  interviews  with  management,  functional,  and  contracting 
officials,  we  tried  to  ascertain  whether,  in  their  view,  large,  multi-func¬ 
tion  or  multi-site  competitions  offered  any  unique  advantages  or  dis¬ 
advantages.  For  the  most  part,  they  said  that  these  types  of 
competitions  offered  a  number  of  distinct  advantages  and  few  disad¬ 
vantages. 

Advantages 

Most  of  officials  we  interviewed  believed  that  large  multi-function/ 
site  competitions  provided  more  flexibility  for  innovation  and  greater 
savings;  a  smaller  in-house  residual  organization  and  lower  overhead; 
clear  command  and  control  accountability;  and  faster,  less  costly  com¬ 
petitions.  One  of  the  principal  advantages  they  cited  was  that  a  single 
organization  with  responsibility  for  everything  was  more  efficient 
than  having  several  organizations  share  responsibility.  The  idea  of 
having  “one  belly-button  to  push”  meant  that  should  problems  arise, 
or  if  changes  needed  to  be  made,  there  was  one  responsible  party 
with  whom  to  communicate.  With  multiple  organizations,  the  need 
for  coordination  would  increase  along  with  the  likelihood  of  bottle¬ 
necks. 

Another  advantage  to  running  multi-function  competitions  is  to 
potentially  lessen  the  impact  on  employees.  If  each  function  or  site  is 
competed  separately,  the  process  would  have  taken  longer,  resulted  in 
duplicate  efforts,  and  been  a  greater  hardship  on  their  employees. 
Competing  each  function  separately  can  result  in  cascading  (reduc¬ 
tion  in  force)  (RIFs)  with  employees  who  have  just  weathered  the 
stress  of  one  competition  finding  themselves  in  a  new  position  about 


43 


to  be  competed.  Another  approach  is  to  run  competitions 
concurrently;  this,  however,  requires  extensive  competitive  sourcing 
staffing. 

Interviews  with  contractors  provided  insight  into  other  advantages. 
Larger,  in  many  cases  multi-function,  competitions  are  of  greater 
interest  to  contractors  than  small  competitions.  The  significant  cost 
associated  with  preparing  and  submitting  a  bid  and  proposal  makes 
smaller  competitions  less  attractive.  Fostering  greater  interest  from 
the  private  sector  promotes  competition  and  therefore  increased  sav¬ 
ings  to  the  government  and  the  taxpayer.  Contractors  also  indicated 
that  multi-function  competitions  provide  opportunities  for  cross-uti¬ 
lization  of  their  labor  force,  particularly  to  meet  surges  in  demand  in 
a  specific  functional  area.  Under  this  business  model,  the  use  of  a 
streamlined  labor  force  can  lead  to  additional  cost  savings  that  are 
not  available  under  a  single-function  study. 

Many  of  these  advantages  have  much  to  do  with  the  approach  used  in 
the  development  of  the  PWS.  For  example,  in  one  multi-site  compe¬ 
tition,  officials  said  that  they  achieved  only  minor  economies  of  scale 
because  separate  work  statements  were  developed  at  each  site  and 
combined  together.  As  a  consequence,  both  the  MEO  and  the  private 
competitors  were  restricted  in  their  approaches  to  one  that  mirrored 
the  existing  process.  Similarly,  they  felt  that  little  time  was  saved  in 
conducting  the  competition,  and  that  the  approach  had  little  impact 
on  improving  quality  of  service  or  how  the  function  was  monitored. 

In  another  competition,  officials  said  that  whereas  a  single  competi¬ 
tion  minimized  the  adverse  impact  on  the  employees,  they  felt  that 
the  use  of  multiple  awards  negated  the  economies  of  scale  that  could 
have  been  achieved  if  the  award  had  been  made  to  a  single  contrac¬ 
tor.  Officials  for  several  competitions  also  said  that  a  series  of  smaller 
competitions  may  have  attracted  specialists  in  each  field  that  could 
result  in  better  performance. 

Disadvantages 

The  officials  we  interviewed  cited  few  obstacles  to  large  multi-func¬ 
tion  or  multi-site  competitions.  The  one  significant  drawback  men¬ 
tioned  is  the  increased  risk  of  non-performance  in  a  multi-function 
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and/or  multi-site  competition.  Through  the  development  of  com¬ 
plete  packaging,  where  all  related  functions  are  provided  by  one  con¬ 
tractor  or  the  MEO,  non-performance  can  have  a  much  greater 
impact  on  mission.  Conversely,  if  only  a  portion  of  a  set  of  related 
functions  are  competed,  in  the  event  of  non-performance,  personnel 
can  be  made  available  to  mitigate  performance  problems.  Multi-func¬ 
tion  competitions  may  also  increase  the  need  for  coordination 
between  other  functions  and  can  limit  opportunities  for  small  busi¬ 
ness  as  prime  contractors.  In  addition,  although  the  fixed  costs  asso¬ 
ciated  with  contracting  officers  and  project  managers  may  remain 
stable  with  larger  contracts,  there  may  not  be  economies  of  scale  asso¬ 
ciated  with  Quality  Assurance  Evaluators. 
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Packaging 

Attributes  of  successful  packaging 

Effective  packaging  is  the  critical  first  step  in  maximizing  competi¬ 
tion.  A  competition  package  can  consist  of  a  single  function  at  a 
single  location,  multiple  functions  at  a  single  location,  or  single  or 
multiple  functions  at  multiple  locations.  Similarly,  a  package  can  con¬ 
sist  of  similar  or  “like”  functions  or  of  dissimilar  of  “unlike”  functions. 

To  package  effectively,  the  individual  (s)  making  the  decision  need  to 
analyze  a  wide  variety  of  factors.  These  factors  typically  include: 

•  Mission  goals  and  targets 

•  Promoting  competition 

•  Structure  and  composition  of  the  private  sector  market 

•  Industry  best  practices 

•  Relevant  labor  and  economic  conditions 

•  Locations  with  the  same  or  similar  functions 

•  Structure  and  composition  of  the  current  government  organi¬ 
zation  and  workforce 

•  Existing  contracts. 

Overall,  packaging  can  be  considered  successful  if  it  promotes  rather 
than  inhibits  competition,  fosters  the  efficient  provision  of  services, 
and  encourages  good  performance.  Packages  should  group  functions 
together  consistent  with  good  business  practices,  ensuring  that  unlike 
or  fragmented  functions  do  not  reduce  or  inhibit  competition. 
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Analysis  of  the  16  competitions 

Much  of  the  focvts  of  our  interviews  with  senior  leadership,  functional 
managers,  and  members  of  each  of  the  16  A-76  teams  was  on  the  rea¬ 
sons,  or  drivers,  for  selecting  a  particular  set  of  functions  for  compe¬ 
tition.  Our  intention  was  to  understand  the  background,  research, 
and  environment  behind  each  competition  package.  This  section 
provides  the  results  of  these  interviews. 

Decision  driver:  meeting  targets 

The  most  common  driver  or  reason  for  selecting  a  particular  set  of 
functions  or  sites  in  a  competition  is  the  need  to  meet  the  FTE  goals 
that  were  established  in  the  Defense  Quadrennial  Review  (DQR) .  In 
half  the  competitions  we  reviewed,  the  officials  said  that  DQR  goals 
influenced  which  functions  were  selected  for  competition.  In  one 
case,  the  functions  were  selected  because  they  were  the  only  ones 
that,  when  added  together,  met  FTE  goals.  In  others,  however,  the 
goals  served  to  force  the  officials  to  examine  all  their  commercial 
functions  and  begin  the  packaging  process. 

Decision  driver:  synergies  from  related  functions 

The  second  most  common  factor  in  the  packaging  decision  is  the 
desire  to  take  advantage  of  the  synergies  that  can  occur  by  packaging 
similar,  or  related,  functions  together.  To  the  extent  that  these  syner¬ 
gies  are  achieved,  the  functions  can  be  provided  more  effectively  at  a 
lower  cost,  regardless  of  whether  the  competition  results  in  contract 
or  in-house  performance.  One  example  of  this  type  of  packaging  is  in 
the  supply  and  transportation  area.  In  one  supply  competition  exam¬ 
ple,  warehousing  was  included  in  the  competition  package  in  order 
to  give  the  resulting  in-house  or  contract  organization  the  ability  to 
better  integrate  warehousing  with  shipping  and  receiving. 

In  a  second  example,  one  headquarters  organization  decided  to  com¬ 
pete  all  the  logistic  functions  except  outbound  transportation  as  a 
single  competition  because  the  functions  were  so  closely  related.  Offi¬ 
cials  we  interviewed  said  that  competing  the  functions  separately 
would  have  taken  longer,  resulted  in  duplicate  efforts,  and  been  a 
greater  hardship  on  the  employees.  Outbound  transportation  was 
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excluded  because  it  was  already  provided  by  firms  such  as  FedEx 
through  GSA  schedule  contracts.  These  schedule  contracts  provide 
the  government  with  an  outbound  transportation  infrastructure.  In 
addition,  some  oversight  responsibilities  such  as  those  of  the  radia¬ 
tion  protection  officer,  accountable  property  officer,  and  personal 
property  officer,  were  excluded  because  it  was  thought  that  these 
functions  were  inherently  governmental. 

Central-  or  headquarters-level  organizations  were  more  likely  to  take 
this  factor  into  consideration  as  compared  to  installation-managed 
competitions  when  making  their  packaging  decisions.  In  four  compe¬ 
titions,  the  headquarters  competition  managers  said  that  they  consol¬ 
idated  all  similar  functions  at  each  location  to  have  an  integrated 
approach  to  their  competitions,  in  order  to  reduce  competition  times 
and  cost  and  take  advantage  of  lessons  learned  from  earlier 
competitions. 

Decision  driver:  meeting  mission  requirements 

Another  primary  factor  in  management  decisions  on  packaging  is  the 
impact  of  a  particular  package  on  the  mission  of  the  organization.  In 
four  competitions,  decision  officials  packaged  their  competitions  in 
order  to  fit  with  their  long-term  mission  goals  or  to  maintain  mission 
capability  at  less  cost. 

Decision  driver:  mitigating  risk 

One  of  the  greatest  concerns  of  a  public  manager  entering  into  an  A- 
76  competition  is  contractor  non-performance.  The  larger  the  com¬ 
petition,  and  the  more  the  functions  included  in  the  package,  the 
greater  the  impact  on  the  mission  in  the  case  of  poor  performance. 
Managers  must  therefore  balance  the  efficiencies  and  synergies  asso¬ 
ciated  with  large  multi-function  or  multi-site  packages  with  the  risk  of 
potential  non-performance  by  a  contractor  or  an  MEO.  In  three  of 
the  competitions  included  in  this  study,  larger,  multi-site  competi¬ 
tions  on  a  regional  or  national  basis  were  explicitly  rejected  because 
headquarters  officials  believed  that  without  prior  experience  in 
public-private  competitions,  the  management  risk  would  be  too 
great. 
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Decision  factor:  evaluating  labor  markets 

None  of  the  16  competitions  included  in  this  analysis  incorporated  a 
full  labor  market  assessment  into  their  packaging  decisions.  By 
understanding  the  labor  market,  better  predictions  can  be  made  as  to 
the  ability  of  contractors  to  hire  the  in-house  workforce,  the  ability  of 
an  MEO  with  a  heavy  pre-competition  military  presence  to  hire  civil¬ 
ians  to  staff  up  to  required  levels,  and  the  overall  availability  and 
accessibility  to  qualified  personnel  in  the  regional  area. 

Most  contractors  indicated  that  one  key  to  the  successful  transition 
from  in-house  to  contract  is  the  capacity  of  the  contractor  to  hire  sig¬ 
nificant  portions  of  the  in-house  workforce.  It  is  therefore,  in  both 
the  government’s  and  the  contractor’s  interest  to  accurately  estimate 
the  number  of  in-house  personnel  interested  in  securing  employ¬ 
ment  with  the  contractor,  if  the  competition  results  in  a  decision  to 
contract.  Employee  age  and  length  of  public  service  play  an  impor¬ 
tant  role  in  whether  contractors  will  be  able  to  hire  large  portions  of 
the  in-house  workforce.  Employees  in  mid-career,  but  under  the  old, 
non-portable,  retirement  system,  have  significant  incentives  to  take 
priority  placement  even  if  it  means  moving  across  the  country,  or 
even  the  world.  In  the  largest  of  the  16  completions — BOS-all  major 
functions  (1) — the  contractor  expected  to  retain  over  half  of  the  in- 
house  workforce.  However,  because  most  of  the  affected  employees 
were  under  the  Civil  Service  Retirement  System  (CSRS)  retirement 
system  and  not  eligible  for  retirement,  most  employees  entered  the 
priority  placement  program.  This  left  the  contractor  understaffed,  in 
a  remote  location,  with  a  scarce  local  labor  pool.  Performance  suf¬ 
fered  in  all  areas. 

In  another  BOS  competition,  information  technology  was  incorpo¬ 
rated  into  the  PWS.  However,  at  the  time  of  competition,  the  local 
and  regional  labor  markets  had  a  dearth  of  information  technology 
professionals.  In  this  case,  the  contractor  had  great  difficulty  staffing 
to  expected  levels  in  this  area  and  performance  suffered. 

Decision  factor:  outsourcing  in  the  private  sector 

In  determining  how  best  to  package  functions  into  a  successful  com¬ 
petition,  an  evaluation  of  how  services  are  provided  in  the  private 
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sector  can  provide  a  valuable  template.  Understanding  what  func¬ 
tions  are  traditionally  out  sourced  by  similar  private  sector  organiza¬ 
tions  can  provide  insight  on  packaging.  For  example,  an  evaluation 
of  the  functions  out  sourced  at  large  universities,  and  how  they  were 
packaged,  can  provide  a  first  step  in  identifying  potential  candidates 
for  competition  within  a  Service  or  Defense  Agency.  This  type  of  anal¬ 
ysis  was  not  used  in  the  packaging  decisions  for  the  16  competitions 
in  our  review. 

Decision  factor:  industry  analysis 

As  mentioned  earlier  in  this  section,  one  measure  of  successful  pack¬ 
aging  is  the  ability  to  attract  private  sector  interest,  or  competition. 
Previous  research  has  shown  a  positive  correlation  between  savings 
rates  and  number  of  bidders.  Therefore,  it  is  important  identify  the 
types  of  firms  one  wants  to  attract,  their  capabilities  in  terms  of  both 
capacity  and  ability  to  provide  specific  services,  and  the  number  of 
firms  in  the  marketplace. 

For  example,  grouping  facilities  management  and  operation  with  an 
information  technology  function  may  hinder  competition  if  few  com¬ 
panies  have  experience,  or  access  to  qualified  subcontractors,  in  both 
fields.  Similarly,  packaging  custodial  functions  into  a  nationwide 
package  may  be  unsuccessful  if  no,  or  few,  national  custodial  firms 
exist. 

Headquarters  officials  of  only  one  competition  examined  the  indus¬ 
try  structure  before  making  its  final  packaging  decision.  Because  of  its 
remote  location,  they  felt  that  a  large  consolidated  competition  was 
the  only  way  they  could  attract  quality  firms.  Their  review  of  the  local 
business  structure  led  them  to  conclude  that  it  was  incapable  of  deliv¬ 
ering  the  needed  service  if  the  functions  were  competed  individually. 
They  also  confirmed  that  there  were  sufficient  large  firms  that  histor¬ 
ically  bid  on  large  packages  of  dissimilar  functions.  They  did  not 
extend  their  review  to  the  current  industry  practices  in  the  various 
functions  to  determine  whether  consolidating  dissimilar  functions 
into  a  single  competition  was  common  practice  and  would  generate 
more  efficiencies  than  packaging  groups  of  similar  functions  into 
separate  competitions. 
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Decision  factor:  inherently  governmental  decisions  and  packaging 
completeness 

In  deciding  the  final  competition  package,  officials  frequently  reex¬ 
amined  all  positions  in  the  functions  and  identified  additional  posi¬ 
tions  that  they  believed  to  be  inherently  governmental  or  otherwise 
exempt  from  competition.  Positions  such  as  employees  who  order  or 
approve  the  purchase  of  supplies,  property  accountability  officials,  or 
electricians  who  perform  intrusion  detection  were  considered  inher¬ 
ently  governmental  and  exempt  from  competition.  In  other  cases, 
activities  such  as  reutilization  and  transfer  and  disposal  of  property 
were  considered  to  be  core  functions  and  were  also  excluded  from 
the  competition.  In  about  half  of  these  cases,  the  officials  we  inter¬ 
viewed  said  that  these  decisions  would  be  reversed  in  the  recompeti¬ 
tion  because  they  found  that  the  functions  were  not  inherently 
governmental  and  that  interspersing  them  within  the  functions 
impeded  work  flow  and  made  accountability  more  difficult. 

In  another  competition,  officials  said  that  savings  would  have  been 
increased  if  all  the  activities  were  included  in  the  competition.  They 
believed  that  it  was  difficult  to  clearly  separate  the  exempted  activities 
from  the  rest  of  the  function  (s)  and  doing  so  made  for  difficult  inter¬ 
faces  with  the  rest  of  the  operations. 

These  decisions  frequently  mean  that  complete  functions  or  activities 
are  not  included  in  the  competition.  It  is  not  uncommon  for  several 
activities  within  a  function  to  be  removed  from  competition.  This  can 
make  the  function  more  inefficient  than  it  was  prior  to  competition. 
If  the  decision  is  to  retain  the  function  in-house,  extra  steps  have 
been  added  to  the  process.  If  the  decision  is  to  contract  out,  the  extra 
steps  can  create  bottlenecks  in  the  process  as  the  contractor  waits  for 
government  approval  to  proceed  or  for  supplies  with  which  to  com¬ 
plete  a  job.  More  importantly,  incomplete  packaging  means  that  the 
contractor  cannot  be  held  accountable  for  the  full  performance  of 
the  function,  and  can  only  be  judged  on  how  well  he  performs  spe¬ 
cific  steps  in  the  process.  It  also  means  that  the  contractor  cannot 
manage  the  function  and  make  substantive  improvements  in  the  pro¬ 
cess.  The  contractor  is  reduced  to  being  an  implementor  of  a 
government  process. 
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Figure  17  illustrates  a  typical  situation  where  the  contractor  is 
accountable  for  work  order  completion  but  does  not  control  work 
order  scheduling,  material  coordination,  or  planning  and  estimation. 
In  this  situation,  the  contractor  cannot  be  held  responsible  or 
accountable  for  timely  job  order  completion  because  he  or  she 
cannot  control  the  full — or  complete — process.  Completeness  in 
packaging  fosters  innovation  and  promotes  effective  and  efficient  ser¬ 
vices,  while  promoting  accountability  and  responsibility  by  the  service 
provider. 


Figure  1  7.  Work  order  process 


•  Closes  out  WO 

•  Maintains  records 


•  Performs  work 

•  Returns  WO  to 
Planning  &  Resource 
Office 


•  Coordinates  with  material 
coordinator  &  shop 

•  Schedules  job  to  shop 


Decision  factor:  small  business  considerations 

Once  the  overall  packaging  decisions  were  made,  several  other  fac¬ 
tors  came  into  play  as  officials  refined  the  final  scope  of  the  competi¬ 
tion.  By  far  the  most  important  secondary  consideration  was  the 
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existence  of  existing  contracts.  With  very  few  exceptions,  functions 
that  were  already  being  performed  by  contract  were  excluded  from 
the  competition.  In  one  case,  decision  officials  originally  included  all 
functions,  whether  or  not  they  were  contracted,  in  order  to  have  a 
single  point  of  accountability  and  increase  potential  synergies.  How¬ 
ever,  many  of  the  contracts  had  been  awarded  to  small  or  disadvan¬ 
taged  firms,  and  the  SBA  objected  to  their  inclusion.  After  about  a  6- 
month  delay  in  the  competition,  the  officials  agreed  to  exclude  all 
small  business  contracts  from  the  scope. 

In  another  case,  the  officials  we  interviewed  said  that  there  was  a  3- 
week  delay  to  the  competition  process  while  they  negotiated  with  SBA 
officials  about  whether  an  existing  contract  could  be  included  in  the 
competition’s  scope.  In  a  third  case,  a  contract  with  the  prison  indus¬ 
tries  was  excluded  from  the  scope  because,  if  contracted  out,  the  con¬ 
tractor  could  not  use  the  existing  contract.  Many  of  the  officials 
involved  in  competitions  that  were  contracted  said  that  subsequent 
recompetitions  would  include  all  relevant  contracts  so  that  there 
would  be  a  single  point  of  accountability. 

Considerations  of  current  small  business  contracts  can  be  viewed  sim¬ 
ilarly  to  the  discussion  of  completeness  in  packaging  for  inherently 
governmental  functions  in  the  section  above.  By  excluding  or  parcel¬ 
ing  out  sections  of  a  complete  activity  or  process,  responsibility  and 
accountability  for  good  performance  are  narrowed  and  opportuni¬ 
ties  for  efficiencies  and  innovation  are  limited. 

The  packaging  decision-maker 

For  the  16  competitions  included  in  this  analysis,  final  decisions  on 
packaging  were  made  at  either  the  headquarters  or  installation  level. 
In  general,  packaging  decisions  made  by  headquarters  officials 
tended  to  include  such  factors  as  the  organizations’s  mission  goals 
and  plans,  the  desire  for  an  integrated  approach,  competing  like 
functions,  and  decreasing  competition  times.  Installation  officials 
placed  a  higher  priority  on  factors  such  as  meeting  the  FTE  targets 
assigned  to  them,  and  retaining  the  current  organizational  structure. 
Packaging  decisions  at  all  levels  were  made  based  on  factors  that  were 
internal  to  the  government  organization.  They  were  not  based  on 
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such  factors  as  the  structure  of  the  private  market  or  current  eco¬ 
nomic  conditions. 

To  promote  successful  packaging,  the  packaging  decision  should  be 
made  at  a  high  enough  level  in  the  organization  to  ensure  that  it  best 
supports  the  organization’s  overall  mission  and  management  priori¬ 
ties.  For  example,  packaging  decisions  made  by  an  installation  may 
reflect  the  best  interest  of  the  base’s  mission  and  its  priority,  but  may 
not  be  in  the  best  interest  of  the  command  or  Service.  We  have  one 
caveat  however.  Headquarters-level  decision-making  should  not  be 
conducted  without  the  input  of  the  installation.  By  including  installa¬ 
tion-level  personnel  in  the  process,  decision-makers  can  be  provided 
with  better  information  on  completeness  in  packaging,  availability  of 
a  qualified  labor  force,  and  potential  synergies  in  selected  functions. 
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Small  business  opportunities 


Large-scale  competitions  have  been  criticized  often  because  they  are 
assumed  to  limit  small  business  opportunities.  It  is  important,  there¬ 
fore,  to  determine  whether  large-scale  competitions  reduce  the 
number  of  opportunities  for  small  business  to  obtain  government 
contracts.  Our  analysis  indicates  that  these  opportunities  are  not  sub¬ 
stantially  decreased  with  large-scale  multi-function,  multi-site 
competitions.  We  found  that: 

•  Six  of  the  16  competitions  were  restricted  to  small  businesses. 

•  Three  additional  contracts  were  awarded  to  small  businesses, 
even  though  the  procurement  was  unrestricted. 

•  Those  contracts  that  were  awarded  to  large  firms  had  signifi¬ 
cant  small  business  goals. 

Small  business  as  prime 

As  mentioned  above,  6  of  the  16  completions  included  in  this  study 
were  restricted  to  small  business.  Of  these  six,  5  resulted  in  conver¬ 
sion  to  contract  with  one  remaining  in-house.  Data  on  the  population 
of  competitions  completed  after  1995  also  indicated  that  30  percent 
of  competitions  with  more  than  100  ETEs  were  restricted  with  about 
half  resulting  in  conversion  to  contract  with  a  small  business  contrac¬ 
tor  as  prime. 

Unrestricted  competitions 

One  common  complaint  from  small  businesses  is  that  unrestricted 
competitions  limit  small  business  involvement.  However,  based  on 
our  analysis  and  discussions  with  the  SBA,  the  Small  and  Disadvan¬ 
taged  Business  Utilization  Office  (SADBU),  functional  managers, 
and  both  large  and  small  contractors,  it  appears  that  this  may  not  be 
the  case.  The  reasons  are  as  follows: 
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Most  unrestricted  competitions  have  significant  small  business 
goals.  For  example,  the  largest  competition  we  reviewed 
required  about  $15  million  of  the  contract  dollars  to  go  to  small 
business,  and  79  percent  of  the  subcontracts  were  to  be 
awarded  to  small  business.  In  addition,  the  prime  contractor 
was  to  mentor  its  subcontractors  and  was  expected  to  reach  out 
to  the  local  business  community  and  help  develop  businesses. 

Using  a  small  business  as  a  sub-contractor,  rather  than  as  prime, 
can  provide  tremendous  opportunities  for  small  business 
growth.  As  a  sub-contractor,  a  small  business  is  not  responsible 
for  large  bonding  requirements  covered  by  the  prime  and  can 
therefore  execute  larger  projects  not  previously  available. 

Mentoring  opportunities  with  large  business  can  provide  small 
businesses  with  leadership,  technology,  and  process  improve¬ 
ments  unavailable  without  such  partnering  arrangements. 

Of  the  171  unrestricted  competitions  completed  since  1995  for 
which  we  have  information  on  contractor  size,  26  competitions 
or  15  percent,  resulted  in  the  prime  contract  being  awarded  to 
a  small  business.  This  indicates  that  unrestricted  competitions 
do  not  create  a  complete  barrier  to  entry  for  small  business. 


Transition 


No  matter  who  is  selected  as  the  new  provider  during  the  competi¬ 
tion — the  government  or  a  contractor — there  is  a  transition  period 
before  the  new  provider  is  fully  operational.  Transitions  set  the  tone 
and  expectations  for  future  performance.  Rocky  transitions  can 
destroy  the  government’s  and  the  customers’  confidence  and  trust 
and  can  take  years  to  overcome.  Often  we  found  that  the  expectation 
is  that  future  performance  will  be  poor.  The  new  provider  frequently 
had  to  perform  at  superior  levels  to  create  the  perception  that  it  is 
performing  satisfactorily.  Transitions  involve  a  shift  in  the  culture, 
structure,  and  operation  of  most  services.  They  take  time  and  effort, 
and  major  roadblocks  can  arise  along  the  way.  A  successful  competi¬ 
tion  will  have  a  comprehensive  plan  to  avoid  the  roadblocks  and  to 
proceed  smoothly. 

Transitions  are  particularly  critical  with  large,  multi-function  or  multi¬ 
site  competitions.  The  importance  of  good  performance  by  the  ser¬ 
vice  provider  becomes  more  critical.  Poor  performance  has  a  much 
greater  impact  on  the  installation’s  mission  than  does  a  small,  single¬ 
function  competition.  Multiple  functions  or  sites  can  add  a  level  of 
complexity  and  coordination  that  may  not  exist  in  single-function  site 
competitions.  Therefore,  it  is  vital  that  transitions  for  large  competi¬ 
tions  be  well  planned  and  managed  by  both  the  government  and  the 
new  provider. 

Of  the  16  competitions  we  examined,  9  had  smooth  transitions  and  7 
had  rocky  transitions.  In  4  of  the  9  smooth  transitions,  the  decision 
was  to  retain  the  function (s)  in-house.  In-house  transitions  were 
smooth,  but  they  proceeded  at  a  more  relaxed  pace  than  did  contrac¬ 
tor  transitions.  About  42  percent  of  the  transitions  to  contract 
performance  were  smooth;  the  rest  faced  difficulties. 

We  found  that  transitions  can  be  rocky  if  they  are  poorly  planned  and 
provide  inadequate  training  and  start-up  time  to  both  government 
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and  contract  employees.  Rocky  transitions  were  also  characterized  by 
poor  relationships  and  differing  expectations  between  the 
government  and  contractor  personnel;  difficulties  in  hiring  sufficient 
numbers  of  experienced  personnel;  backlogged  work;  and  incidents 
such  as  hiding  or  mislabeling  keys  to  buildings  and  equipment.  Of 
these  factors,  the  following  appear  to  be  most  significant  in  the  16 
competitions  studied: 

•  The  relationship  between  the  installation  and  the  new  provider 
is  a  key  indicator  of  how  successful  the  transition  will  be.  If  the 
competition  was  charged  with  tension  and  stress,  it  often  car¬ 
ries  over  to  the  transition  to  contract.  If  the  contractor  isn’t 
sensitive  to  these  tensions  and  focuses  on  allaying  fears  of  poor 
performance,  the  relationship  can  quickly  become  strained. 

•  For  the  16  competitions,  there  was  no  correlation  between  the 
length  of  the  transition  and  the  size  or  complexity  of  the  com¬ 
pletion. 

•  Neither  the  government  nor  the  contractor  has  placed  suffi¬ 
cient  emphasis  on  comprehensive  transition  planning.  The 
government’s  primary  focus  is  on  placing  adversely  affected 
employees,  often  at  the  expensive  of  sustaining  operations.  The 
contractor’s  primary  focus  is  on  the  logistics  of  transition — hir¬ 
ing,  security  clearances,  and  inventory — often  at  the  expense  of 
becoming  familiar  with  the  operations  he  or  she  is  assuming. 

Relationship  between  the  installation  and  the  new  provider 

Every  competition  that  had  a  rocky  transition  was  characterized  by  a 
strained  relationship  between  the  installation  management  and  the 
new  provider.  In  all  seven  of  these  cases,  the  new  provider  was  a  con¬ 
tractor.  Tensions  and  anxiety  can  run  high  during  a  competition.  If  a 
contractor  wins  the  competition,  these  tensions  can  spill  over  into  the 
transition  and  sour  the  relationship  between  the  contractor  and  the 
installation.  However,  a  poor  relationship  is  not  a  foregone  conclu¬ 
sion.  From  our  review,  it  appears  that  the  real  determinant  of  the 
nature  of  the  relationship  is  the  installation’s  leadership.  The  instal¬ 
lation  leadership  sets  the  tone  and  expectations  for  the  competition 
and  the  transition  to  contract  performance.  In  every  case  we 
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examined  where  the  relationship  was  good,  the  installation 
commander,  the  project  manager  for  the  contractor,  and  those 
responsible  for  the  transition  took  an  active  role.  Candid  and  open 
communication  reduced  tensions.  During  the  transition,  installation 
leaders  sent  a  clear  message  that  it  was  not  a  “them  against  us”  situa¬ 
tion,  and  that  the  mission  still  had  to  be  maintained  and  customers 
had  to  be  served. 

Two  competitions  are  particularly  illustrative  of  this  point.  Both  com¬ 
petitions  were  conducted  by  the  same  headquarters  organization. 
The  functions  related  to  supply  that  were  being  competed  were  virtu¬ 
ally  identical,  with  the  operations  at  one  installation  being  somewhat 
larger  and  more  complex  than  the  other.  The  same  contractor  won 
both  competitions.  In  the  smaller  competition,  the  transition  was 
smooth;  in  the  larger,  the  transition  was  rocky.  A  key  difference 
between  the  two  transitions  was  the  leadership  at  the  installations.  In 
the  case  of  the  successful  transition,  the  commander  motivated  his 
employees  by  holding  monthly  “all  hands”  meetings  to  keep  them 
informed;  provided  them  with  cross-training  so  that  whatever  the  out¬ 
come,  the  employees  would  be  more  competitive;  and  emphasized 
that  they  still  had  to  provide  quality  service  to  their  customers.  He 
made  an  effort  to  establish  a  productive  working  partnership  with  the 
contractor,  and  as  a  consequence,  contract  and  government  officials 
we  interviewed  said  that  they  were  able  to  work  through  issues  and 
resolve  problems  quickly. 

At  the  other  installation,  the  commander  opposed  the  competition 
and  would  not  meet  with  the  contractor  or  schedule  the  transition.  As 
a  consequence,  the  contractor  had  difficulty  entering  the  installation, 
identifying  government  employees  to  interview  for  jobs,  and  obtain¬ 
ing  needed  security  clearances.  According  to  the  contractor’s  repre¬ 
sentative,  he  had  to  hire  temporary  employees  to  do  the  inventory 
because  the  government  employees  were  not  allowed  to  assist  the 
contractor.  Once  installation  personnel  knew  that  the  decision  was  to 
contract  out,  they  slowed  their  performance,  thus  creating  a  backlog. 
The  turnover  was  scheduled  to  take  place  at  the  end  of  the  fiscal  year 
and  a  long  holiday  weekend.  As  a  result,  there  was  a  heavier  than 
normal  workload  on  the  first  day  of  the  operation  that  was  difficult  for 
the  contractor  to  overcome.  The  in-house  workforce  also  made  it 
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difficult  for  the  contractor  to  locate  stock  in  the  warehouse,  and 
installation  security  eventually  had  to  be  used  to  find  it.  A  “tiger  team” 
of  in-house  personnel  from  another  location  eventually  had  to  be 
brought  in  to  assist  the  contractor  for  a  period  of  time.  Many  of  these 
transition  problems  could  have  been  avoided  or  minimized  if  the 
installation  leadership  had  been  more  positive. 

In  competitions  where  the  leadership  does  not  support  the  competi¬ 
tion  or  where  the  relationship  between  the  installation  and  contrac¬ 
tor  is  poor,  it  is  not  uncommon  for  the  affected  employees  to 
undermine  the  incoming  contractor.  During  our  review  of  the  16 
competitions,  we  were  told  of  a  number  of  such  incidents.  For  exam¬ 
ple,  in  one  competition,  one  of  the  government  managers  collected 
all  the  keys  to  buildings  and  equipment  but  did  not  turn  them  over  to 
the  contractor.  The  contractor  had  to  call  the  previous  managers  who 
found  the  keys  for  him.  In  another  competition,  depot  stock  was  mis¬ 
placed  and  it  took  installation  security  to  finally  locate  it.  In  a  third 
competition,  underground  communications  cables  were  cut  and  man 
hole  covers  damaged. 

In  addition  to  these  types  of  obstructions,  it  is  not  uncommon  for  the 
installation  workforce  to  make  it  difficult  for  the  contractor  to  enter 
the  installation,  be  slow  in  identifying  employees  to  interview,  or 
create  bottlenecks  in  the  processing  of  paperwork  for  clearances  and 
the  like.  These  obstacles  make  it  difficult  for  the  contractor  to  per¬ 
form  in  a  timely  manner  and  reinforce  the  poor  relationship  with  the 
installation. 

Length  of  transitions 

There  was  no  correlation  between  the  size  or  number  of  functions 
and  the  amount  of  time  allowed  for  the  transition.  Although  contrac¬ 
tors  generally  were  held  to  the  transition  periods  stipulated  in  the 
solicitation,  MEOs  typically  were  not  subject  to  rigorous  time  lines  to 
reach  full  performance. 

For  the  16  competitions,  the  range  for  converting  from  in-house  to 
contract  performance  was  from  60  to  135  days,  with  an  average  tran¬ 
sition  period  of  94  days.  There  was  no  correlation  between  the  length 
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of  a  transition  and  the  size,  number,  or  complexity  of  the  functions 
being  transferred  to  the  new  provider.  For  example,  the  transition 
period  for  the  largest  competition  (more  than  2,000  FTEs  and  15 
major  functions)  was  90  days,  whereas  the  transition  for  a  much 
smaller  function  (120  FTEs  and  7  functions)  was  120  days.  However, 
the  length  of  the  transition  period,  although  it  contributed  to  rocky 
transitions,  did  not  seem  to  be  the  driving  factor.  For  example,  a  com¬ 
petition  of  facilities  maintenance,  transportation,  and  supply  func¬ 
tions  with  234  FTEs  had  a  180-day  transition  period  that  was 
considered  rocky  whereas  a  competition  of  facilities  maintenance, 
transportation,  supply,  and  aircraft  maintenance  functions  with  1,034 
FTEs  had  a  transition  period  of  similar  length  that  was  considered 
smooth. 

Transition  planning 

Neither  the  government  nor  the  contractors  were  particularly  suc¬ 
cessful  in  planning  for  transition.  Of  the  16  competitions,  only  5  were 
well  planned.  Government  and  contractor  officials  for  several  compe¬ 
titions  acknowledged  that  they  had  no  formal  transition  plans  or  said 
that  though  transition  plans  were  developed,  they  were  not  imple¬ 
mented.  Moreover,  if  the  decision  was  to  retain  the  function  (s)  in- 
house,  transition  requirements  in  the  solicitation  were  not  applied  to 
the  MEO.  We  also  found  that  the  government’s  main  focus  during  a 
transition  was  on  placing  the  affected  employees  in  other  jobs,  and 
less  attention  was  paid  to  maintaining  mission  requirements. 

A  comprehensive  government  transition  plan  should  ensure  that: 

•  Mission  requirements  are  met  during  the  transition. 

•  Personnel  are  realigned  and  trained  for  their  new  responsibili¬ 
ties.  Adequate  training  is  particularly  important  if  they  are  to 
assume  quality  assurance  positions  or  are  part  of  the  in-house 
monitoring  efforts. 

•  Inventories  are  taken  of  the  equipment  and  supplies  to  be 
transferred  to  the  new  provider. 


63 


•  Arrangements  for  security  clearances  are  made  where 
necessary. 

•  Customers  are  informed  of  any  changes  in  services  or  proce¬ 
dures  for  obtaining  services. 

Meeting  mission  requirements 

When  evaluating  the  16  competitions,  it  appeared  that  for  the  major¬ 
ity  of  the  competitions,  the  installation’s  main  priority  during  transi¬ 
tion  was  the  placement  of  affected  employees  and  that  little  emphasis 
was  placed  on  maintaining  its  operations  to  sustain  mission  require¬ 
ments.  As  a  consequence,  workload  backlogs  were  frequent  occur¬ 
rences  as  government  employees  were  reassigned  to  other  positions. 
The  backlogs  made  the  new  provider’s  transition  and  its  ability  to  fully 
meet  performance  requirements  more  difficult. 

In  two  competitions,  officials  arranged  for  employees  from  other 
installations  to  supplement  the  in-house  workforce  until  the  new  pro¬ 
vider  was  at  full  performance.  They  also  transferred  some  of  the  work¬ 
load  to  other  installations  until  the  new  provider  was  up  to  speed.  In 
two  other  competitions,  officials  arranged  for  soon-to-be  transferred 
military  personnel  to  supplement  the  remaining  in-house  workforce 
until  the  new  provider  was  operational.  The  military  in  one  competi¬ 
tion  also  monitored  the  transition  and  were  available  to  fill  in  on  an 
emergency  basis  if  needed.  This  type  of  transition  planning  promotes 
not  only  the  uninterrupted  provision  of  services  but  fosters  good  will 
and  partnering  between  the  contractor  and  the  government. 

Realigning  and  training  personnel 

As  part  of  a  transition  to  contract,  in-house  employees  will  often  be 
reassigned  to  the  residual  in-house  organization  that  will  monitor  the 
new  provider’s  performance.  In  a  number  of  the  competitions 
affected,  employees  were  not  reassigned  to  their  new  positions  until 
just  before  the  new  provider  began  performance.  Even  when  the 
people  were  notified  of  their  new  positions  in  advance,  they  could  not 
assume  their  new  responsibilities  until  the  new  provider  started.  The 
employees  we  interviewed  who  were  assigned  to  these  positions  said 
that  the  training  they  had  received  was  minimal  and  inadequate  to 
perform  their  new  jobs.  Employees  required  additional  training 
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particularly  in  the  areas  of  evaluating  contract  performance,  under¬ 
standing  the  scope  of  the  PWS,  and  identifying  specific  contract 
requirements.  They  said  that  it  often  took  6  months  to  a  year  before 
they  felt  comfortable  enough  to  adequately  evaluate  the  contractor’s 
performance. 

Taking  inventory  and  requesting  security  clearances 

The  government  did  make  plans  for  taking  inventories  and  request¬ 
ing  security  clearances.  With  one  exception,  inventories  were  taken 
without  problem.  For  one  supply  competition,  the  contractor  had  to 
forgo  the  inventory  because  of  significant  difficulties  in  hiring  quali¬ 
fied  employees. 

Obtaining  security  clearances  was  a  problem  in  only  one  competition. 
According  to  the  government  officials  we  interviewed,  the  contractor 
did  not  prepare  the  necessary  paperwork  in  a  timely  manner.  As  a 
consequence,  the  government  had  to  supplement  the  contractor 
workforce  with  two  employees  who  worked  part-time  to  perform  cer¬ 
tain  hazardous  materials  duties  until  the  contractor  was  able  to  make 
the  necessary  arrangements,  about  a  month  later. 

Informing  customers  of  new  procedures  and  services 

The  transition  plan  of  only  one  competition  included  developing  a 
guide  for  customers  to  explain  the  changes  in  procedures  and  ser¬ 
vices.  The  guide  explained  the  changes  that  would  occur  regardless 
of  result,  identified  the  responsibilities  of  all  parties,  and  explained 
process  flows  for  the  two  scenarios — conversion  to  the  MEO  or  the 
conversion  to  contract. 

Contractor/MEO  transition  plans 

The  potential  contractor  or  MEO  has  to  develop  its  own  transition 
plans.  These  plans  typically  cover  hiring  needed  employees,  becom¬ 
ing  familiar  with  the  new  operations,  and  transferring  the  inventory 
of  supplies  and  equipment  from  the  pre-MEO  government  opera¬ 
tion.  They  should  also  cover  any  needed  actions  to  ensure  that  they 
will  be  able  to  meet  all  performance  requirements  at  the  end  of  the 
transition  period. 
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MEO  transition  plans 


The  transition  to  an  MEO  is  much  less  traumatic  and  problematic 
than  the  transition  to  a  contract.  Even  though  the  in-house  proposal 
calls  for  changes  in  the  way  the  operations  are  performed,  the  MEO, 
as  the  incumbent,  is  already  familiar  with  the  operations.  The  largest 
obstacles  tend  to  be  learning  to  perform  the  operations  with  fewer 
people  or  with  people  who  have  a  different  skill  mix.  To  the  extent 
that  large  numbers  of  military  personnel  performed  the  function 
prior  to  competition,  the  MEO  may  have  difficulty  hiring  additional 
personnel,  especially  if  the  installation  is  in  a  remote  location,  or  if 
there  is  a  shortage  of  qualified  labor  in  the  local  labor  market.  In  the 
competitions  we  reviewed,  the  transitions  to  an  MEO  were  typically 
longer  than  the  transitions  to  contract,  were  considered  smooth  by 
management,  and  illustrated  problems  in  the  ability  to  hire  at  full 
performance. 

In  two  competitions,  transition  plans  were  developed  but  not  fol¬ 
lowed.  The  individual  who  developed  the  plans  left  before  the  MEO 
was  implemented  and  his  replacement  was  not  able  to  oversee  the 
transition.  One  official  we  interviewed  said  that  a  number  of  pro¬ 
cesses  outlined  in  the  in-house  proposal  had  yet  to  be  implemented. 
In  another  competition,  the  transition  took  months  longer  than  the 
time  line  identified  in  the  solicitation.  In  this  case,  the  installation 
found  it  difficult  to  hire  qualified  employees  and  were  short  about  15 
people  by  the  time  it  was  scheduled  to  be  at  full  performance. 

Contractor  transition  plans 

Many  of  the  contract  representatives  we  interviewed  said  that  in  ret¬ 
rospect,  they  did  not  do  a  good  job  in  developing  their  transition 
plans  or  in  following  the  plans  once  developed.  Two  areas  were  par¬ 
ticularly  problematic — becoming  familiar  with  the  new  operations 
and  hiring  sufficient  numbers  of  qualified  employees. 

Becoming  familiar  with  operations.  The  transition  to  contract  can  take 
place  either  before  or  after  the  contract  start  date.  The  competitions 
we  reviewed  were  about  evenly  split  in  whether  they  required  the  tran¬ 
sition  to  occur  before  or  after  contract  start.  The  competitions  we 
reviewed  typically  had  3  to  7  months  between  contract  award  and 


contract  start,  and  up  to  10  months  between  the  actual  decision  date 
and  contract  start.  To  the  extent  that  the  transition  starts  after  the 
start  date  and  the  relationship  with  the  installation  is  good,  the  new 
provider  can  use  this  time  to  become  familiar  with  the  operations  he 
or  she  is  assuming.  Even  if  the  transition  is  to  occur  prior  to  contract 
start,  a  proactive  provider  often  has  several  months  before  the 
transition  officially  starts  that  can  be  used  to  good  advantage. 

Contractors  in  three  competitions  took  advantage  of  this  extra  time 
as  well  as  the  transition  period  to  become  familiar  with  the  operations 
they  were  assuming.  They  were  particularly  good  examples  of  how 
good  transition  planning  can  result  in  successful  performance  and  a 
strong  relationship  with  the  installation.  In  one  case,  the  contractor 
arrived  early  and  used  the  time  between  the  decision  date  and  the,  for 
example  start  of  the  transition  to  accomplish  administrative  tasks  (set¬ 
ting  up  office  space,  for  example)  and  began  to  familiarize  himself 
with  operations.  Within  two  weeks  of  the  decision  date,  this  contrac¬ 
tor  had  his  senior  staff  in  place  and  had  completed  the  necessary 
paperwork  to  obtain  clearances.  The  contractor  used  the  transition 
period  to  interview  on-site  personnel  and  make  job  offers.  He  offered 
key  people  jobs  at  or  above  the  salaries  they  were  making  as  govern¬ 
ment  employees.  In  this  way,  he  was  able  to  retain  essential  expertise 
that  allowed  him  to  have  a  more  productive  workforce  than  would 
have  been  possible  if  he  had  had  to  hire  new  employees. 

In  another  competition,  three  contractors  were  awarded  four  con¬ 
tracts.  (One  contractor  was  awarded  two  contracts.)  The  familiariza¬ 
tion  process  for  their  transition  was  well  thought  out.  During  the  first 
2  months,  two  of  the  three  contractors  who  won  the  competition 
began  their  transition  by  watching  the  government  perform  the  work. 
During  the  second  month,  the  two  contractors  performed  the  work 
and  the  government  watched.  Total  responsibility  was  turned  over  to 
the  contractors  after  the  government  determined  that  the  contrac¬ 
tors  were  capable  of  performing  effectively.  The  turnover  in  these 
cases  proceeded  ahead  of  schedule.  The  third  contractor  took  over 
immediately  because  the  program  officials  wanted  a  turn-key  transi¬ 
tion.  Both  the  contractor  and  the  program  officials  were  satisfied  with 
the  quality  of  the  contractor’s  performance. 
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These  cases,  however,  were  the  exception  in  the  16  competitions  we 
reviewed.  More  commonly,  the  contractor’s  key  management  staff 
would  arrive  at  the  start  of  the  transition  and  use  most  of  the  transi¬ 
tion  time  to  interview  and  hire  people.  The  full  complement  of  work¬ 
ers  would  not  be  present  until  the  contractor  was  expected  to  be  at 
full  performance.  As  a  consequence,  they  had  no  time  to  become 
familiar  with  the  operations  or  to  be  trained  in  systems  or  procedures 
that  might  be  unique  to  the  operation.  This  also  meant  that  any 
needed  security  clearances  could  not  be  processed  in  advance  of  the 
full  performance  date.  More  importantly,  it  meant  that  the  contrac¬ 
tor’s  successful  performance  during  the  critical  first  months  was 
dependent  on  the  contractor’s  success  in  hiring  experienced  former 
government  employees  who  were  already  familiar  with  the  opera¬ 
tions.  To  the  extent  the  contractor  was  successful  in  his  hiring,  the 
transition  was  deemed  a  success.  If  he  was  unsuccessful,  his  perfor¬ 
mance  suffered  and  the  relationship  with  the  installation  became 
strained. 

Hiring  new  employees 

Most  contractors  competing  in  large-scale  multi-function  or  multi-site 
competitions  do  not  have  a  pre-existing  workforce  to  put  in  place  if 
they  win  the  competition.  Instead,  they  depend  on  hiring  govern¬ 
ment  employees  who  are  being  displaced  from  their  jobs.  Many  of  the 
contractors  we  interviewed  said  that  they  expected  to  hire  at  least  50 
percent  of  the  government  workforce  and  would  have  preferred  to 
hire  all  of  the  displaced  workforce. 

Nine,  or  75  percent,  of  the  contractors  in  the  16  competitions  said 
that  they  had  not  been  able  to  hire  as  many  displaced  governments  as 
they  had  hoped  and  had  difficulty  hiring  qualified  people.  Many  of 
these  contractors  still  have  hiring  difficulties,  especially  if  they  are  in 
remote  locations  or  in  tight  or  limited  labor  markets.  The  contract 
officials  we  interviewed  said  that  they  frequently  have  to  import  work¬ 
ers  in  order  to  augment  the  local  labor  force.  This,  in  turn,  can 
increase  their  turnover  rate  which  is  often  disruptive  and  can 
threaten  performance. 

Several  factors  increase  a  contractor’s  difficulties  in  hiring  displaced 
government  workers.  They  include: 


•  Government  placement  of  affected  workers.  To  the  extent  that 
the  government  is  successful  in  placing  its  affected  workers  in 
other  government  jobs,  the  contractor’s  pool  of  potential 
employees  is  reduced.  Contractors  we  spoke  to  said  that  the 
government  is  often  the  only  source  of  trained  employees  and 
the  extraordinary  efforts  it  exerts  to  place  workers  can  cause 
the  contractor  transition  and  performance  difficulties.  If  the 
contractor  cannot  retain  a  significant  portion  of  the  in-house 
workforce,  the  contractor  will  probably  spend  much  of  the  tran¬ 
sition  period  trying  to  hire  employees  instead  of  using  that  time 
to  familiarize  the  contract  staff  with  the  government  operation. 

•  Reluctance  to  pay  workers  above  the  Department  of  Labor 
wage  rates  for  a  particular  type  of  worker.  Contractors  are  often 
reluctant  to  pay  more  than  DOL  wage  rates  because  doing  so 
may  hurt  their  competitive  position.  To  the  extent  that  contrac¬ 
tors’  offers  do  not  match  or  exceed  government  employees’ 
current  salaries,  the  employees  will  reject  the  contractor’s  offer. 
For  example,  in  two  competitions  the  winning  contractors 
offered  the  displaced  government  workers  less  than  their  cur¬ 
rent  salaries  when  other  government  organizations  in  the 
immediate  area  were  matching  or  exceeding  their  salaries.  As  a 
consequence,  the  displaced  employees  accepted  offers  from 
the  other  government  organizations.  In  fact,  many  of  the 
affected  workers  were  unhappy  that  they  were  placed  within  the 
remaining  government  organization  and  not  RIF’d  so  that  they 
could  transfer  to  the  other  government  organizations  that  were 
hiring. 
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Quality  assurance 


During  our  review  of  the  16  competitions,  we  focused  on  how  the  gov¬ 
ernment  was  performing  its  quality  assurance  responsibilities.  Perfor¬ 
mance  monitoring  is  central  to  a  successful  competition.  The 
government  must  continually  evaluate  the  contractor  or  MEO’s  per¬ 
formance  to  ensure  that  it  meets  the  requirements  established  during 
the  competition.  Under  the  provisions  of  A-76,  the  government  is  to 
ensure  that  the  contractor  or  MEO’s  quality  control  program  is  ade¬ 
quate  to  ensure  full  performance.  Therefore,  it  is  essential  that  the 
individual(s)who  are  assigned  this  responsibility  are  trained  in  quality 
assurance  and  are  technically  knowledgeable  about  the  function.  As 
with  the  members  of  the  Source  Selection  Evaluation  Board  (SSEB), 
it  is  imperative  that  they  be  fair  and  bring  an  objective  attitude  to  the 
job.  Overzealous  monitoring  can  quickly  erode  the  trust  and  respect 
needed  between  the  government  and  the  new  service  provider. 

Quality  assurance  is  essential  to  accurately  measuring  performance. 
If  the  quality  assurance  evaluators  (QAEs)  don’t  know  how  to  evaluate 
contractor  or  MEO  performance,  or  if  they  are  biased  or  overzealous, 
the  actual  performance  may  be  misrepresented.  Therefore,  it  is 
important  that  QAEs  are  well-trained,  objective,  and  measure  whether 
the  provider  meets  the  performance  requirements  in  the  solicitation. 
In  our  review  of  the  16  competitions,  we  found  that: 

•  QAEs,  are  generally  performing  quality  control,  not  quality 
assurance,  and  expect  the  provider  to  perform  the  function  as 
it  was  performed  prior  to  competition. 

•  QAEs  are  not  sufficiently  trained  in  how  to  perform  quality 
assurance,  or  on  the  specific  contract/MEO  performance 
requirements. 

•  Quality  issues  are  frequently  related  to  the  lack  of  fully  devel¬ 
oped  performance  metrics.  For  example,  QAEs  typically  rated 
contractor  performance  of  facilities  operation  and 
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maintenance  functions  lower  than  his  or  her  performance  of 
other  functions  such  as  supply  and  transportation. 

•  In-house  performance  did  not  receive  the  same  scrutiny  as  con¬ 
tractor  performance.  Quality  assurance  for  in-house  wins  was 
characterized  by  a  lack  of  guidance,  inconsistent  implementa¬ 
tion,  and  self-monitoring. 

One  competition,  though  an  extreme  example,  illustrates  what  can 
occur  if  QAEs  perform  quality  control,  are  not  well-trained,  or  are 
overzealous.  In  this  case,  officials  we  interviewed  said  that  QAEs  and 
functional  managers  were  hostile  to  contract  performance  and  rated 
the  contractor’s  performance  very  low.  Although  the  QAEs  had  taken 
the  basic  quality  assurance  course,  they  had  little  knowledge  of  the 
contract  requirements.  In  many  cases,  they  performed  100-percent 
inspections  when  the  surveillance  plan  called  for  inspecting  a 
random  sample.  Within  the  first  6  months,  the  QAEs  had  not 
approved  any  of  the  contractor’s  requests  for  payment.  About 
$890,000  were  in  dispute.  As  a  consequence,  the  contracting  officer 
had  to  intervene  to  break  the  stalemate  between  the  two  parties.  He 
found  that  virtually  none  of  the  contract  discrepancies  claimed  by  the 
QAEs  were  warranted  and  authorized  full  payment  to  the  contractor. 
When  the  monitoring  problems  were  corrected,  the  contractor’s  per¬ 
formance  ratings  improved.  Today,  all  the  officials  we  interviewed 
who  are  involved  in  to  this  competition — including  the  QAEs — 
acknowledge  that  the  contractor  is  currently  meeting  or  exceeding 
the  contract  requirements.  The  functional  manager  said  that  the  con¬ 
tractor  is  performing  the  work  better  than  the  pre-competition  work¬ 
force  in  many  areas. 

Quality  control  versus  quality  assurance 

Circular  A-76  calls  for  the  government  to  perform  quality  assurance 
on  both  contract  and  MEO  performance.  It  also  makes  a  distinction 
between  quality  control  and  quality  assurance.  In  essence,  quality 
control  is  the  day-to-day  monitoring  of  performance.  It  is  typically 
done  at  the  task  level  and  is  the  provider’s  responsibility.  Quality 
assurance  is  the  evaluation  of  whether  the  provider  is  meeting  its 
overall  performance  requirements  and  is  the  government’s 
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responsibility.  In  other  words,  the  distinction  between  quality  control 
and  quality  assurance  is  analogous  to  the  distinction  between  require¬ 
ments  and  performance-based  PWSs.  Quality  control  measures  how 
the  provider  delivers  the  required  service,  whereas  quality  assurance 
measures  ivhether  the  provider  satisfies  the  performance  standards  or 
goals  in  the  solicitation. 

We  found  that  the  QAEs  in  the  competitions  we  reviewed  examined 
the  provider’s  day-to-day  performance,  rather  than  evaluating 
whether  the  provider  met  performance  requirements  or  whether 
there  were  systemic  weaknesses  in  the  provider’s  actions  or  processes 
that  produced  poor  performance. 

One  competition  that  is  currently  in  its  second  year  of  performance 
stands  out.  In  this  case,  the  transition  period  was  shortened  by  3 
months  because  the  in-house  workforce  had  shrunk  to  the  point  that 
backlogs  were  mounting.  Officials  we  interviewed,  including  the  con¬ 
tracting  officer,  said  that  the  QAEs  were  given  minimal  training  and 
lacked  the  skills  necessary  to  communicate  with  contractors,  and 
needed  to  improve  their  writing  skills.  They  also  said  that  the  QAEs 
were  unfamiliar  with  the  contract  requirements  and  the  surveillance 
plan,  and  how  and  when  to  write  deficiency  reports.  Some  of  the 
QAEs,  particularly  those  assigned  to  the  facilities  maintenance  func¬ 
tion,  expected  100-percent  compliance  and  issued  contract  defi¬ 
ciency  reports  for  minor  issues  such  as  a  worker  arriving  late  at  a  job 
site.  As  a  consequence,  there  were  about  12  QAEs  to  monitor  a  con¬ 
tract  workforce  of  90  people. 


Training 


Ideally,  QAEs  should  be  trained  to  perform  quality  assurance,  to  be 
knowledgeable  about  the  solicitation  or  contract  requirements,  and 
to  know  how  to  establish  productive  relationships  with  the  service 
provider.  Typically,  the  QAEs  for  the  16  competitions  we  examined 
were  given  a  basic  quality  assurance  course  either  just  before  or  just 
after  contract  start.  In  general,  they  received  no  formal  training  on 
the  contract  requirements  or  on  how  to  relate  to  the  service  provider. 
Most  commonly,  the  QAEs  were  expected  to  learn  these  aspects  of 
their  new  responsibilities  on  the  job.  The  QAEs  we  interviewed  said 
that  they  needed  more  training  to  perform  their  jobs. 
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They  also  wished  they  could  have  begun  their  jobs  prior  to  contract 
start  so  that  they  could  have  familiarized  themselves  with  their 
responsibilities  and  known  how  to  proceed.  The  most  common  prac¬ 
tice  was  to  establish  the  in-house  organization  monitoring  the  con¬ 
tract  as  close  to  the  contract  start  as  possible.  In  this  way,  the 
installation  could  keep  the  new  QAEs  in  their  former  jobs  as  long  as 
possible  to  ensure  program  continuity  and  have  the  maximum  time 
available  to  place  its  adversely  affected  employees.  However,  this  prac¬ 
tice  frequently  exacerbated  the  difficulties  facing  the  new  QAEs.  It 
was  not  uncommon  that  they  spent  the  first  months  in  their  new  jobs 
trying  to  find  office  space,  furniture,  and  equipment,  rather  than 
familiarizing  themselves  with  the  terms  of  the  contract  and  the  con¬ 
tractor’s  performance. 

Relationship  between  performance  metrics  and  quality 
assurance — an  evaluation  of  facilities  maintenance  and 
operations 


As  discussed  in  the  performance  section  of  this  study,  the  lack  of 
adequate — or  full — performance  metrics  can  create  a  situation 
where  there  is  discord  between  the  government’s  expectation  of  good 
performance  and  the  contractor’s  aspiration  to  meet  the  metrics 
defined  in  the  PWS.  If  metrics  are  ill-defined,  government  personnel 
can  view  contractor  performance  as  lacking,  even  though  the  con¬ 
tractor  is  meeting  the  requirements  and  metrics  outlined  in  the  PWS. 
From  the  contractor’s  perspective,  the  government  is  requesting  a 
standard  of  quality  or  additional  activities  which  are  outside  the  orig¬ 
inal  scope  of  the  contract. 

This  problem  is  particularly  acute  in  the  area  of  facility  operations 
and  maintenance.  Although  16  competitions  are  too  small  a  sample 
to  permit  statistically  meaningful  trends,  we  did  observe  that  QAEs 
frequently  ranked  contractors’  performance  of  facilities  mainte¬ 
nance  and  operations  services  lower  than  they  ranked  their  perfor¬ 
mance  of  other  functions  such  as  supply  and  transportation  services. 
This  was  true  even  if  the  same  contractor  was  performing  all  of  the 
services. 
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When  we  examined  this  further,  we  found  that  QAEs  monitoring 
facilities-related  functions  were  apt  to  measure  the  contractor’s  per¬ 
formance  based  on  how  the  in-house  workforce  had  performed  the 
function.  If  the  contractor  was  not  duplicating  the  in-house  process, 
the  QAE  was  more  likely  to  believe  that  his  or  her  work  was  not  satis¬ 
factory.  In  contrast,  QAEs  monitoring  supply  or  transportation  ser¬ 
vice  tended  to  measure  performance  based  on  specific  performance 
measures  such  as  receipts,  issues,  inventory  accuracy,  and  vehicle  out- 
of-commission  rates.  Because  they  had  clearer  performance 
measures,  they  were  less  likely  to  focus  on  the  process  used  by  the 
contractor. 

Contract  versus  MEO  monitoring 

The  four  competitions  that  remained  in-house  receive  less  day-to-day 
scrutiny  than  do  the  12  competitions  that  were  converted  to  contract 
both  in  terms  of  transition  as  well  as  day-to-day  performance  monitor¬ 
ing.  As  discussed  in  the  transition  section  of  this  report,  government 
organizations  were  slower  to  implement  their  MEOs.  Several  had  not 
reached  their  MEO  levels  or  had  staffed  the  MEO  with  higher  graded 
employees  than  originally  proposed;  and  began  their  formal  monitor¬ 
ing  well  after  the  MEO  began  performance. 

In  three  of  the  four  competitions,  the  installation  had  not  reached  its 
proposed  MEO  level  or  had  not  staffed  the  MEO  at  the  grade  levels 
proposed  in  the  in-house  proposal.  In  one  of  the  competitions,  the 
MEO  had  not  been  reached  and  budget  cuts  at  the  installation 
reduced  the  authorized  staffing  to  below  MEO  level.  There  was  no 
corresponding  reduction  in  workload  documented,  and  it  was 
unclear  whether  the  MEO  was  still  being  held  to  the  performance 
requirements  in  the  solicitation.  In  another  competition,  the  MEO 
reached  its  proposed  staffing  level  of  50  people  for  only  a  brief 
period.  It  is  currently  operating  with  44  people.  In  the  third  competi¬ 
tion,  although  the  staffing  level  proposed  in  the  MEO  had  been 
reached,  it  deviated  from  the  grade  structure  proposed  in  the  MEO. 

Typically,  monitors  have  been  assigned  to  evaluate  the  MEO’s  compli¬ 
ance  with  the  solicitation’s  performance  requirements.  As  a  general 
rule,  these  monitors  have  not  received  the  training  provided  QAEs 
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who  monitor  contractor  performance.  In  at  least  two  competitions, 
the  installation  was  unaware  of  the  requirement  to  monitor  in-house 
performance  until  several  months  after  the  MEOs  were  to  be  at  full 
performance.  Even  after  monitoring  began,  it  was  done  on  a  sporadic 
basis  whenever  the  monitors  could  take  time  from  their  normal 
duties.  The  monitors  we  interviewed  said  that  they  relied  on  the  PWS, 
found  the  quality  assurance  surveillance  plan  (QASP)  hard  to  inter¬ 
pret,  and  did  not  know  the  relative  priorities  between  the  various  per¬ 
formance  requirements.  When  questioned,  the  monitors  did  not 
know  who  received  the  performance  information  they  collected  or 
how  it  was  used.  In  the  third  competition,  functional  managers  self¬ 
monitor  the  performance  of  their  MEO  units.  They  rely  heavily  on 
the  command’s  performance  evaluation  process  which  consists  of 
customer  surveys,  field  interviews,  and  failure  or  rejection  rates. 

Although  the  number  of  monitors  assigned  to  a  competition  is  not 
necessarily  indicative  of  the  quality  of  the  monitoring,  it  is  interesting 
to  note  that  in  competitions  where  the  decision  was  to  retain  the  func¬ 
tion  in-house,  only  one  or  two  monitors  were  assigned  to  evaluate 
MEO  performance.  In  contrast,  in  competitions  won  by  contractors, 
it  is  not  uncommon  to  have  20  or  more  QAEs.  On  the  surface,  it 
would  appear  that  installations  place  less  importance  on  monitoring 
in-house  performance  than  they  do  on  contract  performance. 

When  these  observations  are  taken  in  combination,  it  is  difficult  to 
determine  whether  the  MEOs  in  these  competitions  are  meeting  the 
performance  requirements  contained  in  their  respective  solicita¬ 
tions.  If  DoD  is  to  know  whether  its  functions  are  being  performed 
efficiently  and  effectively,  it  is  essential  that  MEOs  be  monitored  to 
the  same  standards  as  are  contractors  and  that  monitors  of  in-house 
performance  be  given  the  same  quality  assurance  training  as  QAEs. 

A  change  in  performance  monitoring — a  case  study 

After  a  particularly  rocky  transition  that  was  characterized  by  animos¬ 
ity  between  the  functional  organization  and  the  contractor,  hostile 
and  overly  critical  QAEs,  inadequate  training,  and  delays  in  payment 
to  the  contractor,  the  contracting  officer  replaced  the  QAEs  with 
quality  assurance  specialists  (QASs)  from  another  organization.  The 
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QASs  were  experienced  in  overseeing  large  contracts;  had  additional 

IQ 

training  including  ISO  training;  and  brought  a  different  and  more 
neutral  approach  to  contract  monitoring.  Instead  of  inspecting  every 
task,  their  approach  was  to  evaluate  the  contractor’s  quality  control 
program  and  ensure  that  it  was  resulting  in  quality  performance.  If 
there  seemed  to  be  a  performance  problem,  they,  together,  with  the 
contractor,  would  determine  whether  changes  were  needed  to  the 
quality  control  program  so  that  the  problem  or  similar  problems 
would  not  recur.  Because  they  were  separate  from  the  functional 
area,  they  were  able  to  be  more  neutral  in  their  evaluation  of  the  con¬ 
tractor’s  performance.  They  were  also  able  to  focus  more  on  the 
results  the  contractor  achieved,  and  less  on  the  processes  he  used  to 
achieve  those  results.  All  of  the  officials  we  interviewed  said  that  this 
approach  is  working  well  and  that  there  is  a  productive  relationship 
between  all  parties  now.  They  said  that  they  much  preferred  this 
approach  to  their  earlier  one  and  are  now  using  it  on  all  public- 
private  competitions. 


13.  ISO  refers  to  an  international  set  of  quality  management  standards  that 
have  been  developed  over  time  by  a  network  of  national  organizations 
focused  on  process  management. 
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Ease  of  competition 


Finally,  to  determine  whether  large  competitions  were  easier  or 
harder  to  conduct  than  small,  single-function,  single-site  competi¬ 
tions,  we  examined  not  only  the  length  of  time  it  took  to  conduct  the 
competition,  but  each  step  of  the  competition  process  as  well. 

A  successful  A-76  competition  is  structured  and  conducted  in  such  a 
way  to  ensure  access  to  the  widest  number  of  competitors,  one  of 
whom  will  be  selected  and  evaluated  objectively.  To  attract  the  widest 
number  of  competitors,  the  organization  must  package  the  competi¬ 
tion  effectively  and  describe  the  work  to  be  done  in  terms  of  the 
results  to  be  achieved,  not  the  processes  by  which  it  is  to  be  done.  The 
descriptions  of  the  work  are  generally  referred  to  as  performance 
work  statements  (PWSs) .  Competitors  must  also  be  assured  that  their 
proposals,  and,  if  selected,  their  work  will  be  evaluated  objectively. 
Effective  transitions  selection  and  quality  assurance  are  the  keys  to 
the  successful  performance  of  the  new  provider. 

The  same  competition  process  is  used  whether  the  competition  is 
large  or  small.  We  found  that,  for  the  most  part,  competition  size  or 
complexity  does  not  affect  the  ease  of  competition.  As  discussed 
throughout  this  study,  several  steps — packaging,  transition,  and  qual¬ 
ity  assurance — are  very  important  in  deciding  whether  to  conduct  a 
large  multi-function  or  multi-site  competition  and  in  determining  its 
ultimate  success. 

The  remaining  steps — PWS  and  MEO  development,  contracting,  and 
independent  review14 — while  important  components  of  an  A-76  com¬ 
petition,  are  not  necessarily  made  harder  or  easier  by  the  size  or  com¬ 
plexity  of  the  competition.  Overall,  we  found  that: 


14.  OMB  has  recently  proposed  a  number  of  revisions  to  the  A-76  process. 
If  these  revisions  are  adopted,  the  requirement  to  conduct  an  indepen¬ 
dent  review  of  the  in-house  proposal  will  be  eliminated. 
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•  PWSs  were  becoming  more  performance-based  than  those  we 
have  previously  reviewed,  but  they  still  needed  improvement. 

•  MF.Os  generally  did  not  propose  major  changes  to  the  current 
operations,  rather  they  reduced  overhead  and  down-graded 
positions. 

•  Independent  reviewers  in  all  but  one  competition  found  no  sig¬ 
nificant  errors  in  the  in-house  proposal.  In  several  competi¬ 
tions,  however,  independent  reviews  expressed  concern  that 
they  were  not  experts  in  all  of  the  functional  areas  and  believed 
that  they  were  at  a  disadvantage  when  evaluating  functional 
capabilities. 

•  The  contract  type  was  not  always  well  matched  to  the  level  of 
performance  risk  associated  with  the  functions. 

•  Small  business  received  a  significant  amount  of  BOS  work. 

We  highlight  our  specific  findings  for  these  steps  in  this  section.  A 
more  detailed  examination  of  each  step  is  provided  in  appendix  A. 

PWS  and  MEO  development 

When  we  examined  the  PWS  development  in  the  16  competitions,  we 
found  that: 

•  The  PWSs  ranged  from  outcome-based  to  requirements-based. 
Seventy-five  percent  were  still  requirements-based,  but  more 
approached  being  performance-based  than  previous  competi¬ 
tions  we  have  reviewed.  One  competition  was  outcome-based,  a 
first  for  A-76  competitions. 

•  Seven  competitions,  or  44  percent,  used  generic  PWs;  the 
remainder  developed  unique  PWSs. 

—  Generic  PWSs  offered  consistency  among  competitions  and 
saved  overall  development  time.  For  the  most  part,  officials 
we  interviewed  were  pleased  with  generic  PWSs  unless  they 
limited  flexibility  to  adapt  to  different  workloads  or  accom¬ 
modate  the  unique  aspects  of  their  functions. 
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—  Unique  PWSs  were  more  likely  to  be  requirements-based  or 
prescriptive.  Local  officials  were  pleased  with  their  unique 
PWSs  because  they  described  the  functions  in  greater 
detail,  but  in  hindsight  they  wished  that  they  had  been 
more  performance-based  so  that  the  new  provider  could  be 
more  innovative. 

•  PWSs  and  workload  estimates  were  often  outdated  by  the  time 
the  new  provider  began  work. 

With  respect  to  the  development  of  the  in-house  proposal  or  MEO, 
we  found  that: 

•  The  majority  of  the  competitions — 62.5  percent — used  local 
teams  to  develop  the  MEO. 

•  The  MEOs  generally  did  not  make  significant  changes  in  the 
pre-competition  work  processes,  rather  they  reduced  costs  by  a 
“winning”  percentage  by  reducing  overhead  and  downgrading 
positions. 

•  Efforts  were  made  to  establish  firewalls  between  the  PWS  and 
MEO  teams. 

Independent  review 

Overall,  the  independent  review  of  the  16  competitions  found  no  sig¬ 
nificant  errors  in  the  in-house  proposals.  There  was  one  exception  to 
this  overall  finding.  In  that  case,  the  problems  with  the  in-house  pro¬ 
posal  were  major  and  the  proposal  had  to  be  significantly  revised.  In 
general  competition  officials  were  satisfied  with  the  quality  of  the 
independent  review.  Even  though  there  was  overall  satisfaction,  sev¬ 
eral  source  selection  officials  for  several  of  the  competitions  found 
instances  where,  in  their  opinion,  the  in-house  proposal  did  not  meet 
the  solicitation  requirements.  In  several  competitions,  independent 
reviews  expressed  concern  that  they  were  not  experts  in  all  of  the 
functional  areas  and  believed  that  they  were  at  a  disadvantage  when 
evaluating  functional  capabilities. 


81 


Contracting  considerations 

Although  all  of  the  competitions  we  reviewed  were  negotiated  pro¬ 
curements,  there  was  a  mix  of  contract  types  and  restricted  and  unre¬ 
stricted  procurements,  and  differing  methods  to  evaluating  best  value 
and  the  in-house  proposals.  Specifically,  we  found  that: 

•  The  contract  type  was  not  always  well  matched  to  the  level  of 
performance  risk  associated  with  the  functions. 

•  Small  business  received  a  significant  amount  of  BOS  work: 

—  Thirty-one  percent  of  the  competitions  were  set  aside  for 
small  businesses. 

—  An  additional  12  percent  were  awarded  to  small  business, 
though  the  competition  was  unrestricted. 

—  Large  businesses  had  substantial  small  business  goals. 

•  The  majority,  62  percent,  of  the  source  selection  authorities 
(SSAs)  were  from  headquarters  organizations. 

•  The  in-house  proposals  were  evaluated  in  a  variety  of  ways.  In 
several  competitions,  the  source  selection  evaluation  boards 
(SSEBs)  evaluated  the  proposals  as  they  would  a  contractor’s 
proposal. 

•  Although  a  number  of  competitions  received  appeals  or  pro¬ 
tests,  and  in  one  case,  a  court  challenge,  none  of  these  actions 
reversed  the  initial  decision. 
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Appendix  A:  Competition  process 

PWS  and  MEO  development 

PWS  and  MEO  development  are  two  of  the  most  work-intensive  steps 
in  the  A-76  process.  The  PWS  defines,  for  both  the  in-house  and  pri¬ 
vate-sector  competitors,  what  is  being  requested,  the  performance 
standards  and  measures,  and  time  frames  required.  It,  more  than  any 
other  step,  determines  the  quality  and  number  of  bidders  and  the 
maximum  level  of  performance  the  organization  can  expect.  The 
MEO  represents  the  government  workforce’s  opportunity  to  be  more 
competitive  than  the  private  sector.  It  is  vitally  important  that  both 
steps  produce  quality  products. 

Overall,  we  found  that  the  PWSs  were  becoming  more  performance- 
based  than  those  we  have  previously  reviewed,  but  they  still  needed 
improvement.  We  found  that  MEOs  still  needed  to  be  more  innova¬ 
tive  and  competitive. 

When  we  examined  the  PWS  development  in  the  16  competitions,  we 
found  that: 

•  The  PWSs  ranged  from  outcome-based  to  requirements-based. 
Seventy-five  percent  were  still  requirements-based,  but  more 
approached  being  performance-based  than  previous  competi¬ 
tions  we  have  reviewed.  One  competition  was  outcome-based,  a 
first  for  A-76  competitions. 

•  Seven  competitions,  or  44  percent,  used  generic  PWS;  the 
remainder  developed  unique  PWSs. 

—  Generic  PWSs  offered  consistency  among  competitions  and 
saved  overall  development  time.  For  the  most  part,  officials 
we  interviewed  were  pleased  with  generic  PWSs  unless  they 
limited  flexibility  to  adapt  to  different  workloads  or  accom¬ 
modate  the  unique  aspects  of  their  functions. 
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—  Unique  PWSs  were  more  likely  to  be  requirements-based  or 
prescriptive.  Local  officials  were  pleased  with  their  unique 
PWSs  because  they  described  the  functions  in  greater 
detail,  but  in  hindsight  they  wished  that  they  had  been 
more  performance-based  so  that  the  new  provider  could  be 
more  innovative. 

•  PWSs  and  workload  estimates  were  often  outdated  by  the  new 
provider  began  work. 

With  respect  to  the  development  of  the  in-house  proposal  or  MEO, 
we  found  that: 

•  The  majority  of  the  competitions — 62.5  percent — used  local 
teams  to  develop  the  MEO. 

•  The  MEOs  generally  did  not  make  significant  changes  in  the 
pre-competition  work  processes,  rather  they  reduced  costs  by  a 
“winning”  percentage  by  reducing  overhead  and  downgrading 
positions. 

•  Efforts  were  made  to  establish  firewalls  between  the  PWS  and 
MEO  teams. 

PWS  development 

In  examining  the  development  of  the  performance  work  statement, 
we  focused  on  four  key  areas.  We  also  attempted  to  determine 
whether  there  is  any  correlation  between  these  factors  and  costs,  per¬ 
formance,  and  how  the  competition  was  managed.  The  four  areas 
are: 


•  Performance-based  versus  prescriptive  PWSs 

•  Generic/ template  versus  unique  PWSs 

•  Accuracy  and  currency  of  PWS’s  and  workload  estimates 

•  Quality  of  performance  measures  and  standards. 

Performance  versus  requirements  based  PWSs 

In  our  report  on  the  long-run  costs  and  performance  of  competitive 
sourcing,  we  found  that  the  PWSs  we  reviewed  were  typically  very 
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requirements-based  and  restricted  bidders  to  duplicating  the  current 
work  processes.  We  also  found  that  older  PWSs  were  more  restrictive 
than  the  more  recent  ones.  Since  the  16  competitions  we  reviewed  for 
this  study  were  more  recent  than  those  in  the  previous  study,  we  were 
curious  to  determine  whether  the  quality  of  PWSs  had  become  more 
performance-based  over  time.  The  following  chart  depicts  where 
each  PWS  is  on  the  continuum  from  requirements  to  outcome-based. 
We  have  included  outcome-based  PWSs  in  the  continuum  because, 
though  not  required  by  A-76,  one  competition  in  our  sample 
developed  an  outcome-based  PWS. 

Figure  1  8.  Type  of  PWS 
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|Range  Operations!  |BOS-all  major  functions  (2)| 


Requirements  Based: 

&  Ease  of  costing 
&  Limits  performance  risk 
&  Completeness  of  contingencies 
Little  opportunity  for  innovation 

Limits  opportunities  for  savings  through  limiting  competition 

Performance  Based: 

&  Promotes  efficiency  and  innovation 

&  Places  responsibility  and  accountability  for  good  performance  on  service  provider 
fc  Focuses  on  the  end  product  rather  than  prescribing  how  to  conduct  the  work 

Outcome  Based: 

&  Allows  the  maximum  amount  of  innovation  and  efficiency 
&  Maximizes  opportunity  for  savings 
Increased  performance  risk 

‘I  Lack  of  specificity  may  make  costing  difficult  if  workload  data  inadequate 
Lack  of  specificity  may  lead  to  problems  meeting  installation  “expectations" 


Generic  versus  unique  PWS’s 

Government  officials  can  either  tailor  a  PWS  to  each  individual  com¬ 
petition  or  develop  a  generic  PWS  prior  to  the  individual  competition 
that  can  be  applied  to  the  competition  once  it  has  begun.  Generic 
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PWSs  provide  a  template  for  individual  competitions  which  establish 
a  standard  scope  of  work  for  the  function  (s)  while  allowing  for  some 
modification  in  order  to  incorporate  unique  requirements.  Each  type 
of  PWS  has  distinct  advantages  and  disadvantages. 

Generic  PWSs  can  save  competition  time.  After  the  PWS  is  developed, 
it  can  be  applied  to  all  competitions  of  the  same  function.  They  pro¬ 
vide  consistent  performance  measures  and  goals  which  make  assess¬ 
ing  performance  across  multiple  locations  easier  for  headquarters 
organizations.  They  also  provide  contractors  with  a  consistent  set  of 
rules  if  they  bid  on  multiple  competitions  of  the  same  function. 
Generic  PWSs  also  make  identifying  and  applying  “lessons  learned” 
to  subsequent  competitions  easier. 

However,  if  generic  PWSs  don’t  adequately  allow  for  the  incorpora¬ 
tion  of  an  installation’s  unique  requirements,  performance  can 
deteriorate  and  make  it  difficult  fulfill  the  terms  of  the  solicitation. 
Finally,  there  is  an  increased  risk  that  specific  activities  are  uninten¬ 
tionally  omitted  from  the  PWS  in  the  drive  to  standardize  perfor¬ 
mance  from  one  location  to  another.  This  risk  is  enhanced  if  local 
officials  who  are  responsible  for  the  function’s  day-to-day  operation 
are  not  part  of  the  development  effort. 

Unique  PWSs  have  the  advantage  of  being  specifically  tailored  to  the 
given  set  of  functions  covered  by  a  competition.  They  provide  the 
opportunity  to  have  a  more  comprehensive  scope  of  work  that  accu¬ 
rately  reflects  all  individual  activities  that  should  be  performed.  They 
are  particularly  appropriate  if  the  function  being  competed  is 
unique.  Typically,  it  takes  more  time  to  develop  separate  PWSs  for 
each  competition  than  it  would  if  one  PWS  were  developed  for  a  func¬ 
tion  that  is  performed  at  multiple  locations. 

For  example,  the  storage  of  excess  or  surplus  aircraft  may  be  unique 
to  a  single  location,  whereas  facilities  operation  and  maintenance  are 
performed  at  every  location.  Therefore,  the  PWS  for  aircraft  storage 
will  by  definition  be  unique,  while  a  PWS  for  facilities  operation  and 
maintenance  can  be  developed  for  each  individual  location,  or  devel¬ 
oped  once  for  application  to  all  locations.  A  unique  PWS  for  the  facil¬ 
ities  operation  maintenance  function  at  each  location  will  take  more 
time  to  develop  than  a  generic  PWS  for  application  of  all  sites.  In 
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addition,  there  is  the  risk  that  the  individual  PWSs  will  vary  greatly  in 
quality  and  have  vastly  different  performance  standards  and  goals. 
The  differences  between  multiple  PWSs  for  essentially  the  same  func¬ 
tion  can  cause  needless  confusion  and  delay. 

Of  the  16  competitions  we  examined,  all  were  for  functions  that 
existed  at  multiple  locations.  Seven,  or  44  percent,  used  generic 
PWSs.  A  unique  PWS  was  developed  for  each  of  remaining  9  compe¬ 
titions.  All  of  the  competitions  using  generic  PWS’s  were  centrally 
managed  by  headquarters  organizations.  Five  centrally-managed 
competitions  developed  unique  PWSs.  All  of  the  installation-man- 
aged  competitions  developed  unique  PWSs.  (See  figure  17) 

Figure  19.  Generic  versus  unique  PWS 


Competition 

Generic 

Unique 

Type  of  management 

Length  of 
competition 

BOS-transportation,  supply,  and  material 
maintenance 

X 

Installation 

37 

BOS-all  major  functions  1 

X 

HQ 

29 

Range  operations 

X 

26 

Target  operations 

X 

38 

BOS-all  major  functions  2 

X 

HQ  managed/ 
installation  performed 

22 

BOS-supply  &  transportation 

X 

HQ  managed/ 
installation  performed 

28 

Supply  1 

X 

HQ 

21 

Supply  2 

X 

HQ 

23 

Facilities  operation  and  maintenance 

X 

Installation 

24 

Information  management 

X 

Installation 

23 

BOS-all  major  functions  3 

X 

HQ 

27 

BOS-all  major  functions  4 

X 

HQ 

38 

Communications 

X 

Installation 

22 

Property  disposal 

X 

HQ 

22 

For  the  most  part,  the  headquarters  officials  we  interviewed  believed 
that  generic  PWSs  were  very  effective.  Local  officials  mirrored  that 
view,  if,  in  their  opinion,  there  was  sufficient  flexibility  to 
accommodate  unique  aspects  of  their  functions  or  widely  different 
workload  levels.  They  appreciated  the  use  of  a  generic  PWS  because 
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they  believed  that  they  did  not  have  sufficient  expertise  to  develop  an 
adequate  PWS  on  their  own.  To  the  extent  that  the  generic  PWS 
offered  limited  flexibility,  however,  local  officials  were  more  critical. 
In  one  competition,  officials  repeatedly  stated  that  the  PWS  did  not 
adequately  describe  the  large-scale  deployments  that  occurred  at 
their  installation.  As  a  consequence,  they  believed  that  quality  of  work 
had  suffered  and  had  on  occasion  threatened  their  readiness. 

Most  local  officials  were  pleased  with  the  quality  of  the  PWSs  they  had 
developed  on  site.  They  believed  that  they  were  sufficiently  detailed 
to  accurately  describe  the  work  to  be  performed.  In  hindsight,  how¬ 
ever,  several  local  officials  said  that,  in  retrospect,  they  wished  that  the 
PWS  had  been  more  performance-based,  because  their  contractors 
could  not  be  innovative  under  the  current  requirements.  The  offi¬ 
cials  said  that  these  problems  would  be  corrected  in  the  recompeti¬ 
tion.  In  one  locally-managed  competition,  officials  expressed  the  view 
that  it  would  have  been  preferable  if  their  headquarters  organization 
had  developed  the  PWS  because  they  were  frustrated  at  the  competi¬ 
tion’s  long  duration  and  their  lack  of  expertise.  They  also  stated  that 
their  regular  duties  suffered  while  they  participated  in  the 
competition. 

Accuracy  and  currency  of  PWS  and  workload  estimates 

The  third  area  we  examined  was  the  accuracy  and  currency  of  the 
PWS  and  the  workload  estimates  it  contained.  Because  of  long  pro¬ 
curement  leadtimes,  it  is  very  difficult  to  make  PWSs  and  workload 
estimates  accurate  and  current.  It  is  not  uncommon  for  the  PWS  to 
reflect  work  requirements  and  conditions  that  are  two  years  old  by  the 
time  the  decision  is  made  to  contract  out  or  retain  the  government 
workforce.  However,  it  is  critically  important  that  the  PWS  adequately 
describe  all  the  functions  that  need  to  be  performed  and  provide 
realistic  workload  estimates.  If  functions  or  portions  of  functions  are 
omitted,  needless  functions  included,  or  workload  faulty,  it  will  be 
impossible  for  the  new  provider,  contract  or  MEO,  to  perform  ade¬ 
quately  within  the  costs  it  proposed.  Poor  PWS  and  workload  esti¬ 
mates  create  frustration  and  dissatisfaction  by  all  parties  and  escalate 
tensions  needlessly  between  the  government  managers  and 
customers  and  the  provider.  This  is  particularly  true  if  there  is  no  easy 
way  to  make  corrections. 
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MEO  development 

Ten  of  the  16  competitions  used  local  teams  to  develop  their  MEOs. 
The  majority  of  officials  we  interviewed  believed  that  the  local  func¬ 
tional  managers  were  in  the  best  position  to  streamline  their  organi¬ 
zation.  Of  the  competitions  whose  MEOs  were  developed  centrally, 
headquarters  officials  said  that  local  officials  are  less  able  to  make 
major  reductions  or  changes  and  risk  increased  tension  and  dissatis¬ 
faction  in  the  workforce.  Forty  percent  of  the  locally  developed  MEOs 
prevailed  in  the  competition;  none  of  the  centrally  developed  MEOs 
prevailed. 

A  wide  variety  of  techniques  were  used  to  develop  the  MEOs  of  the 
competitions  we  examined.  They  included  using  an  MEO  from 
another  competition  as  a  starting  point;  using  a  pre-determined 
reduction  goal  of  30  percent;  process  reengineering;  changing  the 
labor  mix;  and  automation  and  modernization. 

In  one  competition,  the  strategy  was  to  reduce  in-house  costs  by  25  to 
30  percent,  because  the  historical  average  savings  of  winning  in-house 
estimates  is  25  to  30  percent  below  current  costs.  The  consultant 
developing  the  management  plan  used  a  methodology  called  “rapid 
prototyping.”  The  technique  is  used  to  perform  business  process  re¬ 
engineering  and  relies  heavily  on  interviews  with  functional  manag¬ 
ers  to  identify  work  processes  and  workload.  Additional  data  collec¬ 
tion,  analyses,  and  interviews  were  also  necessary  to  develop  and 
validate  the  reasonableness  of  the  MEO. 

With  the  exception  of  two  competitions,  efforts  were  made  to  estab¬ 
lish  firewalls  between  the  PWS  and  MEO  teams.  In  these  two  cases, 
the  same  team  developed  both  the  PWS  and  MEO.  Both  competitions 
were  conducted  in  1999.  At  that  time,  there  was  no  prohibition 
against  using  the  same  people  on  both  teams.  The  firewalls  that  were 
employed  included  such  things  as  maintaining  separate  offices  on 
separate  floors  or  buildings,  and  establishing  separate  password- 
protected  data  files. 

In  one  competition,  the  contracting  office  treated  the  MEO  team  as 
if  it  were  a  private  sector  provider  and  refused  the  team  access  to  the 
PWS  until  the  solicitation  was  released.  In  most  cases,  however,  the 
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MEO  team  had  access  to  a  draft  PWS  as  soon  as  it  was  completed.  And 
in  several  cases,  the  MEO  team  had  access  to  segments  of  the  PWS  as 
they  were  completed  and  were  able  to  begin  preparing  the  in-house 
proposal  before  the  solicitation  was  issued.  MEO  team  members  fre¬ 
quently  said  that  they  needed  this  extra  time  in  order  to  submit  the 
in-house  proposal  on  time. 

Independent  review 

Overall,  the  independent  review  of  the  16  competitions  found  no 
significant  errors  in  the  in-house  proposals.  There  was  one  exception 
to  this  overall  finding.  In  that  case,  the  problems  with  the  in-house 
proposal  were  major  and  the  proposal  had  to  be  significantly  revised. 
In  general,  competition  officials  were  satisfied  with  the  quality  of  the 
independent  review.  Even  though  there  was  overall  satisfaction,  sev¬ 
eral  source  selection  officials  for  several  of  the  competitions  found 
instances  where,  in  their  opinion,  the  in-house  proposal  did  not  meet 
the  solicitation  requirements.  In  several  competitions,  independent 
reviews  expressed  concern  that  they  were  not  experts  in  all  of  the 
functional  areas  and  believed  that  they  were  at  a  disadvantage  when 
evaluating  functional  capabilities. 

The  independent  review  is  the  government’s  last  check  to  ensure  that 
the  in-house  proposal  is  error-free  before  it  enters  into  competition 
with  the  private  sector  proposals.  It  is  designed  to  ensure  that  the  in- 
house  proposal  can  meet  the  PWS  requirements  and  the  cost  estimate 
is  complete,  accurate,  and  complies  with  A-76  requirements.  Circular 
A-76  requires  that  the  independent  reviewer  be  impartial  and  organi¬ 
zationally  separate  from  the  function  being  competed  and  separate 
from  the  organization  preparing  the  in-house  proposal.  The  inde¬ 
pendent  review  must  be  completed  and  the  in-house  proposal  sub¬ 
mitted  to  the  contracting  officer  by  the  proposal  submission  date. 

Results  of  independent  review 

The  independent  reviewers  found  no  significant  problems  with  over 
half  of  the  16  competitions.  Three  in-house  proposals  required  some 
significant  change.  For  example,  in  one  case  the  MEO  team  proposed 
to  reduce  the  number  of  personnel  needed  to  perform  the  functions 
by  closing  a  site.  The  independent  reviewer  believed  that  the  PWS 
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prohibited  any  closures,  and  required  that  the  MEO  be  revised  to 
reflect  his  finding. 

The  independent  reviewer  found  numerous  problems  in  only  one 
competition.  The  problems  were  such  that  the  MEO  had  to  be 
redone  and  the  competition  extended  by  6  months.  In  this  case,  the 
independent  reviewer  found  deviations  from  the  solicitation,  math 
errors,  inconsistencies  within  functions,  insufficient  analyses  to  sup¬ 
port  assumptions,  and  inadequate  historical  performance  backup,  as 
well  as  position  descriptions,  grades,  and  series  that  were  inconsistent 
with  PWS  requirements  and  the  MEO.  These  problems  were  resolved 
by  a  separate  team  from  the  headquarters  organization  who  cor¬ 
rected  documents,  gathered  additional  data,  revised  the  in-house 
proposal,  and  established  audit  trails  for  every  issue. 

Satisfaction  with  independent  review 

For  the  most  part,  the  officials  we  interviewed  were  satisfied  with  the 
independent  reviewer  and  the  quality  of  their  work.  In  several  com¬ 
petitions,  officials  said  that  the  reviewers  needed  additional  training, 
especially  in  the  functions  areas  being  competed.  Also,  contracting 
officials  in  several  competitions  said  that  the  independent  reviewers 
did  not  catch  all  the  problems  in  the  in-house  proposals,  and  that 
they  did  not  meet  the  PWS  requirements.  As  a  consequence,  changes 
had  to  be  made  to  the  in-house  proposal  during  the  selection  process. 

In  at  least  two  cases,  the  officials  believed  that  independent  reviewers 
were  risk  adverse  and  unwilling  to  accept  innovative  approaches  that 
may  have  been  included  in  the  in-house  proposal.  For  example,  in 
one  competition,  members  of  the  MEO  team  said  that  the  indepen¬ 
dent  reviewer  challenged  every  efficiency  proposal  and  made  it  more 
difficult  for  the  in-house  team  to  compete  effectively.  In  their  view, 
the  independent  reviewer  was  unwilling  to  accept  any  amount  of  risk. 
Instead,  the  reviewer  used  historical  workload  and  fund  expenditures 
to  develop  “should  cost”  estimates  to  perform  the  PWS.  All  proposed 
changes  to  the  in-house  operations  were  compared  to  this  baseline. 
These  officials  were  dissatisfied  with  the  independent  reviewer’s 
performance  and  would  prefer  not  to  use  the  reviewing  organization 
for  future  competitions. 
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Who  performed  the  independent  review 

The  independent  reviews  for  all  but  two  of  the  16  competitions,  or  87 
percent,  were  done  by  a  headquarters  organization.  Seven  indepen¬ 
dent  reviews  were  conducted  by  a  Service  audit  agency,  and  seven 
were  conducted  by  financial  analysts  at  the  headquarters  organiza¬ 
tion.  Only  two  were  conducted  at  the  local  level.  However,  the  head¬ 
quarters  organization  has  since  changed  its  policy,  and  future 
competitions  at  these  installations  will  be  performed  by  the  Service 
audit  agency.  Consultants  were  used  in  three  cases.  The  consultants 
performed  the  analysis  for  the  audit  service  or  headquarters  organi¬ 
zation  who  reviewed  the  consultants’  work  and  signed  the 
certification  as  the  independent  review  official. 

For  the  most  part,  the  review  was  performed  by  a  single  person,  usu¬ 
ally  someone  with  financial  or  auditing  experience.  In  competition, 
however,  the  independent  review  was  performed  by  a  team  composed 
of  a  financial  management  analysts  and  representatives  from  human 
resources,  each  functional  area,  and  procurement.  Originally,  a  rep¬ 
resentative  from  the  headquarters  organization  was  to  conduct  the 
independent  review.  However,  he  soon  decided  that  he  needed  addi¬ 
tional  expertise  to  perform  a  comprehensive  review,  where  upon  the 
headquarters  organization  moved  to  a  team  approach.  Using  this 
approach,  each  team  member  reviewed  that  portion  of  the  in-house 
proposal  needing  his  or  her  expertise.  For  example,  the  human 
resources  representative  reviewed  the  MEO  structure  for  appropriate 
grades  and  positions;  the  functional  experts  reviewed  the  feasibility  of 
the  proposed  efficiency  changes  in  their  respective  functions;  and  the 
financial  analyst  reviewed  all  the  costing  information  for  reasonability 
and  accuracy.  The  headquarters  organization  was  so  pleased  with  the 
team  approach  that  it  is  using  it  with  all  subsequent  competitions. 
The  organization  believes  that  this  approach  provides  a  more  com¬ 
prehensive  review  in  a  shorter  timeframe.  The  reviews  are  taking 
approximately  one  week. 

Contracting  considerations 

An  A-76  competition  must  comply  with  the  normal  federal  contract¬ 
ing  process,  including  selecting  a  contract  type,  deciding  whether  the 
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procurement  should  be  restricted  to  small  business,  issuing  a  solicita¬ 
tion,  evaluating  proposals,  selecting  a  winner,  awarding  a  contract, 
and  monitoring  the  contract.  If  the  in-house  proposal  wins  the  com¬ 
petition,  the  MEO  and  associated  technical  performance  plan  (TPP) 
serves  as  the  contract,  and  the  government  must  monitor  its  perfor¬ 
mance  as  if  it  were  a  private  sector  firm. 

Although  all  of  the  competitions  we  reviewed  were  negotiated  pro¬ 
curements,  there  was  a  mix  of  contract  types  and  restricted  and  unre¬ 
stricted  procurements,  and  differing  methods  to  evaluating  best  value 
and  the  in-house  proposals.  Specifically,  we  found  that: 

•  The  contract  type  was  not  always  well  matched  to  the  level  of 
performance  risk  associated  with  the  functions. 

•  Small  business  received  a  significant  amount  of  BOS  work. 

—  31  percent  of  the  competitions  were  set  aside  for  small  busi¬ 
nesses. 

—  An  additional  12  percent  were  awarded  to  small  business, 
though  the  competition  was  unrestricted. 

•  The  majority,  62  percent,  of  the  source  selection  authorities 
(SSAs)  were  from  headquarters  organizations. 

•  The  in-house  proposals  were  evaluated  in  a  variety  of  ways.  In 
several  competitions,  the  source  selection  evaluation  boards 
(SSEBs)  evaluated  the  proposals  as  they  would  a  contractor’s 
proposal. 

•  Although  a  number  of  competitions  received  appeals  or  pro¬ 
tests,  and  in  one  case,  a  court  challenge,  none  of  these  actions 
reversed  the  initial  decision. 

Contract  type  and  risk 

The  16  competitions  we  reviewed  spanned  various  contract  types. 
Ranging  from  firm  fixed-price  contracts — with  and  without  indefinite 
delivery  indefinite  quantity  or  reimbursable  provisions — to  cost-plus 
contracts.  Each  of  these  contract  vehicles  is  appropriate  within  an 


93 


Appendix  A 


A-76  competition.  However,  the  choice  of  which  contract  vehicle  to 
use  can  be  a  factor  in  both  potential  savings  and  contractor 
performance. 

Contract  types  should  be  selected  based  on  the  amount  of  risk  associ¬ 
ated  with  performing  the  contact.  Risk  is  defined  two  ways:  first,  as  the 
risk  to  the  government  of  non-performance  by  the  contractor  for  a 
pre-determined  price  and,  second,  the  risk  to  the  contractor  of  cost 
overruns  without  reimbursement  from  the  government.  Therefore, 
although  firm  fixed-price  contracts  promote  efficiency,  there  is  a 
greater  risk  of  non-performance  at  the  negotiated  price  if  the  scope 
and  workload  are  not  current  or  accurate,  or  if  large  variations  in 
workload  are  expected  to  occur  throughout  the  contract  period.  Con¬ 
versely,  if  the  scope  of  work  is  well  defined  with  only  small  variation  in 
workload  expected,  the  risk  of  large  variation  in  cost  to  both  the  con¬ 
tractor  and  the  government  is  minimal.  Under  this  scenario,  the  use 
of  a  firm  fixed-price  contract  will  promote  efficiency  by  the  contrac¬ 
tor  with  minimal  risk  of  non-performance  or  cost  overruns. 

When  large  variations  in  workload  or  scope  are  expected  or  when 
workload  estimates  in  the  PWS  are  old  or  incomplete,  a  cost-plus  type 
of  contract  is  generally  employed.  Cost-plus  contracts  do  not  provide 
the  same  incentives  as  firm  fixed-price  contracts  to  minimize  total 
cost.  However,  cost-plus  contracts  do  minimize  the  risk  of  cost  over¬ 
runs  to  the  contractor  from  inaccurate  workload  estimates,  thereby 
minimizing  the  risk  of  non-performance. 

If  some  portions  of  the  scope  of  work  are  well-defined,  but  others  less 
so,  it  is  common  to  use  some  combination  of  contract  types,  such  as 
firm  fixed-price  for  the  well-defined  areas  and  a  reimbursable  portion 
for  the  areas  subject  to  more  uncertainty. 

Therefore,  when  the  workload  is  expected  to  be  stable  throughout 
the  contract  period  and  the  workload  and  scope  estimates  are  current 
and  accurate,  there  is  little  performance  risk  and  a  fixed-price 
contract  can  be  used.  On  the  other  hand,  if  the  requirements  are 
complex,  not  well-defined,  or  likely  to  change  substantially  in  the 
future,  and/ or  the  workload  estimates  are  outdated  or  inaccurate, 
the  performance  risk  is  higher.  Because  it  is  difficult  to  price  this  type 
of  work  with  any  confidence,  a  cost-plus  contract  is  more  appropriate; 
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it  protects  both  the  government  and  the  contractor.  Figure  18  shows 
the  relationship  between  the  level  of  risk  and  contract  type. 


Figure  20.  Risk  level  and  contract  type 
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In  addition  to  selecting  a  contract  type,  contracting  officials  also  must 
decide  whether  performance  would  be  enhanced  by  providing  some 
type  of  monetary  incentive  to  the  contract.  In  those  cases,  an  incen¬ 
tive  or  award  fee  can  be  added  to  the  contract  type. 

1 6  competitions,  risk,  and  contract  type 

The  majority  of  the  16  competitions  used  some  type  of  firm  fixed- 
price  contract.  Only  2  competitions  used  a  cost-plus  type  of  contract. 

One  example  that  illustrates  the  level  of  risk  that  can  be  present  in  a 
competition  follows.  The  large  competition  was  managed  by  the 
headquarters  organization  with  little  involvement  by  installation  offi¬ 
cials  in  the  development  of  the  PWS  and  source  selection.  The  PWS 
was  outcome-based,  and  the  specific  performance  standards  and 
goals  were  defined  as  part  of  the  proposal  process.  Because  this  was 
the  first  time  the  functions  were  being  competed,  some  activities  were 
inadvertently  omitted  from  the  solicitation,  and  workload  estimates 
were  based  on  2-year-old  data.  At  the  time  of  the  competition,  the 
installation’s  strategic  importance  was  low,  and  the  expectation  was 
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that  the  work  would  decline  over  the  performance  period  and  the 
installation  would  eventually  close.  As  a  consequence,  the  level  of  per¬ 
formance  described  in  the  PWS  was  lower  than  the  current  level,  with 
workload  estimates  declining  over  time.  Potential  contractors  were 
asked  to  address  how  they  would  scale  down  operations  in  their  pro¬ 
posals. 

By  the  time  of  contract  award,  the  installation’s  strategic  importance 
had  risen  dramatically.  Instead  of  a  steadily  decreasing  workload  and 
eventual  closure,  it  faced  increased  workload  and  the  prospect  of 
additional  functions  being  transferred  to  the  installation.  These 
changes  exacerbated  the  deficiencies  in  workload  estimates  and 
increased  the  performance  expectations  of  installation  personnel 
responsible  for  monitoring  the  contract.  The  contract  was  not  revised 
to  reflect  these  changes  prior  to  award.  As  a  consequence,  the  con¬ 
tractor  found  it  difficult  to  predict  what  needed  to  be  done.  The  firm 
fixed-price  portion  of  the  contract  was  very  difficult  to  modify  to 
reflect  the  changing  reality.  As  a  result,  installation  managers  were 
unhappy  with  contractor  performance  in  areas  critical  to  readiness, 
and  the  contractor  was  forced  to  assume  significant  cost  overruns  to 
meet  the  requirements  of  an  inaccurate  PWS.  A  cost-plus  contract 
would  have  provided  more  flexibility  to  adjust  to  the  changing 
circumstances  and  would  have  mitigated  the  amount  of  risk  of  non¬ 
performance  as  well  the  amount  of  risk  to  the  contractor. 

Composition  of  source  selection  board  and  selection  decision 

A  Source  Selection  Authority  (SSA)  receives  the  recommendations  of 
the  Source  Selection  Evaluation  Board  (SSEB)  and  makes  the  final 
decision  whether  to  contract  out  or  retain  the  function  in-house.  If 
the  decision  is  to  contract  out,  the  SSA  also  selects  the  winning  con¬ 
tractor.  In  10  of  the  16,  or  62  percent,  of  the  competitions,  the  source 
selection  authorities  were  from  headquarters  organizations.  In 
several  cases,  this  was  true  even  though  the  rest  of  the  competition 
had  been  conducted  by  the  installation.  In  those  cases,  the  SSA  was 
the  head  of  the  headquarters  procurement  organization. 

SSEBs  typically  were  composed  of  a  combination  of  headquarters  and 
installation  officials  from  functional,  financial,  and  procurement 
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organizations.  In  two  competitions,  the  headquarters  organization 
decided  that  no  installation  officials  could  be  included  in  the  evalua¬ 
tion  and  selection  process.  However,  this  has  since  been  changed,  and 
functional  experts  are  now  included  in  the  selection  process.  In  two 
of  the  competitions,  customers  were  included  in  the  selection  pro¬ 
cess,  and  in  one  competition,  functional  experts  from  another  instal¬ 
lation  were  included. 

For  the  most  part,  there  was  no  difficulty  in  finding  members  to  serve 
on  the  SSEB.  However,  officials  in  four  competitions  said  they  did 
have  difficulties  in  finding  members.  In  one  case,  it  was  difficult  to 
find  people  who  could  commit  the  needed  time  to  the  selection  pro¬ 
cess.  In  two  competitions,  excluding  installation  officials  limited  the 
pool  of  qualified  candidates.  Officials  in  four  of  the  competitions  said 
that  the  SSEB  members  were  not  adequately  trained  to  evaluate  the 
proposals  and  ascertain  whether  the  potential  providers  could  fulfill 
the  PWS  requirements. 

Evaluation  of  the  in-house  proposal 

There  was  variety  in  how  the  in-house  proposal  was  evaluated.  In  two 
competitions,  the  SSEB  was  not  allowed  to  see  the  in-house  proposal; 
in  at  least  three  competitions,  the  in-house  proposal  was  treated  like 
a  contract  proposal  and  the  MEO  team  was  required  to  make  a  pre¬ 
sentation.  In  two  of  these  cases,  the  SSEB  believed  that  the  in-house 
proposal  did  not  adequately  meet  the  PWS  requirements  and 
required  the  MEO  team  to  make  changes  to  bring  its  proposal  into 
compliance. 

Appeals  and  protests 

Eight  competitions  had  appeals  or  protests.  There  was  one  protest 
and  28  appeals.  In  five  competitions,  there  were  multiple  appeals. 
The  largest  number  of  appeals  for  a  competition  was  14.  The  other 
competitions  with  multiple  appeals  average  about  3  appeals  each. 
There  was  one  court  challenge.  Minor  changes  were  required  in  only 
one  competition  as  a  result  of  appeals.  The  protest  to  the  General 
Accounting  Office  (GAO)  was  not  upheld.  None  of  these  challenges 
changed  the  initial  decision.  The  court  challenge  was  dismissed 
because  it  was  ruled  that  the  union  did  not  have  standing. 
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Appendix  B:  Tobit  regression  model 


To  quantify  the  factors  that  influence  potential  competition  savings, 
we  estimated  a  Tobit  regression  model.  In  this  appendix,  we  present 
the  full  results  of  the  model  discussed  in  the  section  on  savings.  The 
dependant  variable  is  the  percentage  savings  realized  from  the  com¬ 
petition.  We  include  explanatory  variables  to  explore  some  previously 
unexplained  aspects  of  competitive  sourcing  and  variables  categoriz¬ 
ing  competitions  as  single-  or  multiple-function  studies.  We  use  a 
Tobit  model  because  the  percentage  savings  is  censored  from  below 
at  0,  and  is  correct  for  heteroskedasticity. 

Regression  variables 

Table  9  lists  a  description  and  the  mean  value  of  the  variables  used  in 
the  regression  model. 

We  included  only  those  competitions  for  installation  services  in  the 
categories  of  installation  services,  other  non-manufacturing  opera¬ 
tions,  and  maintenance  of  real  property.  These  studies  represent 
about  74  percent  of  all  competitions  conducted  between  1978  and 
January  2002.  About  30  percent  of  the  studies  contained  military  bil¬ 
lets,  and  nearly  two-thirds  of  the  studies  were  competed  at  locations 
that  had  at  least  one  other  competition  previously.  Only  a  small 
number  of  the  competitions  contained  at  least  one  utility — about  5 
percent — and  only  about  3  percent  of  the  studies  did  not  receive  any 
satisfactory  private  sector  bids. 

We  placed  competitions  into  one  of  three  categories:  1)  studies  with 
only  one  function  code,  2)  studies  with  multiple  function  codes,  or  3) 
a  whole  base  competition — designated  with  CA  function  code  PI 00 
in  the  CAMIS  files.  We  group  studies  in  this  manner  to  examine  the 
effects  of  combining  studies  without  making  a  subjective  determina¬ 
tion  as  to  whether  a  study  was  indeed  a  single-  or  multiple-function 
competition.  One  drawback  to  this  approach  is  that  it  limits  the  ability 
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Table  9.  Description  and  mean  value  of  the  variables  used  in  the  regression  model 


Variable  Description  Mean 

RUR_URBA  1  if  competition  is  not  within  an  MSA;  0  otherwise  0.22 

PCTMIL  Ratio  of  military  billets  to  total  billets  competed  0.12 

UNIQ_STS  Number  of  unique  UlCs  1.07 

LNBS  Natural  logarithm  of  number  of  baseline  billets  2.83 

LNBS2  Square  of  the  natural  logarithm  of  number  of  baseline  billets  9.60 

MIL  1  if  competition  contains  military  billets;  0  otherwise  0.30 

PRECOMP  1  if  previous  competition  conducted  at  UIC;  0  otherwise  0.63 

NOSAT  1  if  no  satisfactory  private  sector  offers  were  received;  0  otherwise  0.03 
UTIL  1  if  study  contained  a  utility;  0  otherwise  0.05 

BOS2  1  if  study  contained  multiple  functions;  0  otherwise  0.18 

BOS3  1  if  study  code  was  CA  function  code  PI  00;  0  otherwise  0.02 

COMP2  1  if  study  was  conducted  by  the  Air  Force;  0  otherwise  0.37 

COMP3  1  if  study  was  conducted  by  the  Marine  Corps;  0  otherwise  0.02 

COMP4  1  if  study  was  conducted  by  the  Navy;  0  otherwise  0.38 

COMP5  1  if  study  was  conducted  by  a  Defense  Agency;  0  otherwise  0.03 

SOL2  1  if  study  was  restricted  to  SBA  8(a)  set-aside;  0  otherwise  0.04 

SOL3  1  if  study  was  restricted  to  other  small  business;  0  otherwise  0.01 

SOL4  1  if  study  was  not  restricted  to  small  business;  0  otherwise  0.34 

LN77  Natural  logarithm  of  time  trend  1.76 


to  decompose  the  effect  of  various  combinations  of  functions  on 
potential  competition  savings.  We  expect  that  economies  of  scale  or 
scope  would  be  less  likely  when  combining  functions  that  are  unre¬ 
lated  than  when  combining  functions  that  are  related.  For  example, 
fewer  economies  would  be  available  when  combining  intermediate 
aircraft  (CA  code  J501)  with  family  services  (CA  code  G904)  than 
when  combining  it  with  intermediate  maintenance  of  aircraft  engines 
(CAcodeJ502). 

The  mean  values  indicate  that  most  studies — about  80  percent — 
included  only  one  function  code.  Only  2  percent  of  the  studies  were 
designated  as  PI 00. 


Regression  results 

Table  10  provides  the  full  regression  results  for  the  heteroskedastic 
Tobit  regression  model.  The  coefficients  in  a  Tobit  model  indicate 
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the  expected  effects  on  the  dependant  variable  if  the  censoring  were 
removed.  Since  the  process  that  led  to  the  censoring  still  exists,  we 
examine  the  expected  effects  on  observed  savings  and  calculate  mar¬ 
ginal  values  using  the  regression  coefficients  and  the  correction  for 
heteroskedasticity.  The  coefficients  are  evaluated  at  the  mean  of  the 
other  variables.  Here,  we  discuss  several  key  variables  not  described 
above. 


Table  10.  Full  regression  results 


Variable 

Coefficient 

Standard  error 

P-value 

Constant 

-0.177 

0.042 

0.00 

RURJJRBA 

-0.035 

0.014 

0.01 

PCTMIL 

0.107 

0.032 

0.00 

UNIQ_STS 

-0.008 

0.011 

0.46 

LNBS 

0.152 

0.021 

0.00 

LNBS2 

-0.014 

0.003 

0.00 

MIL 

0.054 

0.018 

0.00 

PRECOMP 

-0.014 

0.014 

0.32 

NOSAT 

-0.150 

0.030 

0.00 

UTIL 

-0.044 

0.024 

0.07 

BOS2 

-0.033 

0.015 

0.03 

BOS3 

-0.054 

0.027 

0.04 

COMP2 

0.042 

0.017 

0.01 

COMP3 

0.041 

0.046 

0.37 

COMP4 

0.000 

0.017 

0.98 

COMP5 

0.070 

0.038 

0.07 

SOL2 

-0.089 

0.029 

0.00 

SOL3 

-0.100 

0.044 

0.02 

SOL4 

-0.012 

0.013 

0.36 

LN77 

0.061 

0.011 

0.00 

Log  likelihood:  -236.18 

Proportion  with  no  savings:. 21 

Satisfactory  private-sector  bids 

The  variable  NOSAT  is  an  indicator  variable  for  those  competitions 
where  no  satisfactory  private-sector  bids  were  received.  This  designa¬ 
tion  in  GAMIS  includes  studies  where  no  bids  or  offers  were  received 
and  instances  where  all  bids  received  were  deemed  not  responsible. 
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The  coefficient  indicates  that  not  receiving  a  satisfactory  private- 
sector  proposal  significantly  reduces  potential  competition  savings — 
by  about  12  percentage  points,  holding  all  else  constant.  This  sup¬ 
ports  previous  findings  that  competition  is  the  main  driver  for 
realizing  savings  during  an  OMB  circular  A-76  competition.  DoD 
should  place  additional  emphasis  on  ensuring  that  each  study  is  pack¬ 
aged  appropriately  and  advertised  sufficiently  to  maintain  private 
sector  interest. 

Small  business 

We  include  several  indicator  variables  to  control  for  the  effects  of 
restricting  a  competition  to  a  small  business.  The  excluded  variable 
group  studies  were  restricted  to  small  businesses.  The  coefficient  on 
SOL4 — the  indicator  variable  for  unrestricted  competitions — is  neg¬ 
ative  but  statistically  insignificant.  This  indicates,  on  average,  compe¬ 
titions  that  were  not  restricted  to  small  business  did  not  realize  higher 
savings  than  those  that  were  restricted.  In  other  words,  the  small  busi¬ 
ness  restriction  did  not  affect  potential  competition  savings. 

Workforce  composition 

The  variable  controlling  for  the  composition  of  the  workforce — MIL 
and  PCTMIL — are  both  positive  and  statistically  significant  which 
indicates  that  both  increase  potential  competitions  savings,  holding 
all  else  constant.  The  coefficient  on  the  proportion  of  the  workforce 
that  is  military  personnel  indicates  that  changing  a  study  from  an  all¬ 
civilian  to  an  all-military  workforce  increases  potential  savings  by 
about  1 1  percent  at  the  mean  of  all  the  other  variables.  Including  at 
least  one  military  billet  in  the  activity  increases  the  percentage  savings 
from  the  competition  by  about  5  percentage  points,  all  else  held 
constant. 
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Appendix  C:  Methodology 

To  estimate  the  cost  savings  from  large,  multi-function,  multi-site  compe¬ 
titions,  we  selected  16  competitions  that  had  been  competed  between 
1996  and  2000.  Our  intent  was  to  be  able  to  review  the  full  and  actual 
costs  to  perform  these  functions  for  at  least  one  year  of  the  solicitation 
period  specified  in  the  A-76  competition. 

This  section  outlines  the  methodology  used  in  our  assessment  of  costs, 
as  well  as  all  caveats  and  assumptions,  and  discusses  all  problems  and  lim¬ 
itations  to  the  data  available.  Our  approach  covers  the  following  steps: 

•  Competition  selection 

•  Data  collection 

•  Data  analysis. 

Competition  selection 

Criteria 


Time  frame:  1996  to  2000 

All  of  the  competitions  we  examined  were  completed  between  1996 
and  2000.  We  selected  this  time  period  for  a  variety  of  reasons  that 
included: 

•  Selection  of  functions  that  were  likely  to  be  prevalent  in  future 
competitions 

•  Review  competitions  using  the  most  recent  revision  of  OMB 
Circular  A-76 

•  Availability  of  the  greatest  number  of  individuals  who  were 
involved  in  the  competition  process; 

•  Availability  of  at  least  one  full  year  of  operational  data  to  review. 
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Competitions  with  100  or  more  FTE’s 

Because  there  was  no  uniform  definition  of  a  Base  Operations  Sup¬ 
port  (BOS)  competition,  we  selected  our  sample  of  16  competitions 
from  multi-function  or  multi-site  competitions  that  have  100  or  more 
FTEs  when  they  were  announced.  However,  it  should  be  noted  that 
subsequent  to  announcement,  several  of  our  sample  decreased  in  size 
and  on  occasion  the  final  number  of  FTEs  was  less  than  100. 

Balance  between  contract  and  in-house  wins 

We  chose  functions  that  had  a  history  of  both  contract  and  in-house 
wins  so  we  could  identify  any  differences  in  cost  and  performance 
trends. 

Functions  that  have  some  relevance  for  future  competitions 

For  the  most  part,  we  selected  competitions  whose  functions  were 
representative  of  the  current  competitive  sourcing  inventory  so  that 
their  examination  would  have  relevance  for  future  competitions. 

Component  representation 

We  selected  competitions  from  each  Service  and  at  least  one  defense 
agency. 

Telephone  screening 

Before  we  visited  an  installation  to  interview  personnel  and  review 
documents,  we  first  called  the  installation  to  ensure  that  the  relevant 
documents  would  be  available.  This  initial  screening  eliminated  sev¬ 
eral  competitions  because  the  appropriate  personnel  were  no  longer 
available  to  be  interviewed  or  because  we  could  not  arrive  at  a  mutu¬ 
ally  satisfactory  time  to  visit  the  site. 

Data  collection 

Installation  interviews 

After  telephone  screening,  we  visited  installations  to  conduct  inter¬ 
views  and  collect  competition  data.  During  these  visits,  we  inter¬ 
viewed  installation  personnel  that  routinely  came  in  contact  with  the 
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function  that  had  been  competed — functional  managers;  quality 
assurance  evaluators;  customers;  and,  if  the  competition  was  a  con¬ 
tract-win,  contracting  personnel  and  the  contractor.  We  used  a  struc¬ 
tured  interview  form  designed  to  gather  information  on  both 
competition  cost  and  performance.  The  goal  of  the  interview  process 
was  to  gain  an  understanding  of: 

•  The  tasks  being  conducted  within  the  function  and  what  tasks, 
if  any,  differ  from  traditional  perceptions  (i.e.,  identifying  any 
unique  characteristics  of  the  function) 

•  The  history  of  the  competition.  Whether  there  were  any 
specific  problems  with  the  cost  comparison  or  competition 
process,  any  protests,  and  the  number  and  types  of  bidders 

•  Any  major  changes  in  how  the  function  was  provided  pre-  and 
post-  competition 

•  The  major  changes  to  workload  during  each  solicitation  period 

•  The  major  changes  to  scope  during  each  solicitation  period 

•  The  major  one-time  cost  changes  during  each  performance 
period 

•  The  quality  of  the  performance  of  the  function  and  whether 
performance  has  changed  during  the  period  of  performance 

•  Any  additional  costs  to  the  government  not  identified  in  the 
cost  comparison  or  contract  documentation. 

Documentation  review 

During  each  site  visit,  we  reviewed  all  available  documentation  on  the 
cost  and  performance  of  the  function.  Documentation  review 
included:  the  competition  documentation  (PWS,  cost  comparison, 
correspondence,  bids,  and  protests)  and  all  post-competition  con¬ 
tracts,  modifications  and  purchase  orders  (if  contract-win),  budgets, 
audit  and  manpower  reports  (if  in-house  win),  and  performance 
reviews.  We  also  obtained  relevant  workload  information  if  it  was 
available. 
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Supplementary  data 

In  addition  to  the  site  visits  and  interviews,  we  obtained  additional 
data  to  verify  and  augment  the  data  collected  at  the  installation.  This 
supplementary  data  included  audit  reports,  data  from  Service-wide 
information  systems,  and  private-sector  cost  data  on  comparable 
functions. 


Data  analysis 


Once  we  had  gathered  cost  and  performance  data  on  the  selected 
competitions,  we  analyzed  them  to  determine  whether  actual  costs 
were  greater  or  less  than  originally  estimated  and  if  performance  met 
the  level  specified  in  the  competition.  The  goal  of  this  analysis  was  to 
examine  whether  the  expected  level  of  savings  from  A-76  competi¬ 
tions  was  achieved  and  maintained  without  impact  to  the  quality  of 
services  provided. 

Tracking  cost  changes 

Pre-  and  post-competition  costs 

To  determine  the  expected  level  of  savings  from  A-76  competitions, 
the  annual  costs  of  a  function  post-competition  must  be  isolated  and 
tracked  over  each  solicitation  and  compared  with  pre-competition 
costs.  To  have  an  accurate  comparison  of  the  full  pre-  and  post-com¬ 
petition  costs,  components  of  cost  must  be  isolated  and  defined. 

•  Pre-competition  costs.  Throughout  this  study,  comparisons  are 
made  between  a  function’s  annual  costs  and  its  pre-competi¬ 
tion,  or  baseline  costs.  In  the  past,  estimates  of  baseline,  costs 
have  come  under  criticism,  stating  that  baseline  cost  estimates 
per  billet  are  too  high.  To  address  these  concerns,  we  have  used 
the  MEO  cost  per  billet  (as  reported  in  the  cost  comparison) 
and  applied  this  ratio  to  the  pre-competition  billets  to  estimate 
baseline  costs.  This  provides  a  more  conservative  estimate,  and 
assumes  that  the  cost  per  billet,  both  pre-and  post  competition, 
is  the  same. 
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•  Post-competition  costs.  Post-competition  costs  include  the  total 
direct  cost  of  providing  the  service  plus  any  indirect  costs  to  the 
government.  The  direct  cost  of  providing  a  service  is  the  con¬ 
tract  price  in  the  case  of  a  contract  win,  the  cost  of  meeting  the 
MEO  for  an  in-house  win.  Indirect  costs  include  contract 
administration  costs,  one-time  conversion  costs  (amortized 
over  the  first  solicitation  period),  and  any  other  costs.  All 
calculations  of  post-competition  cost  include  both  the  direct 
and  indirect  cost  of  providing  the  function. 

Tracking  post-competition  costs 

The  total  costs  estimated  on  the  cost  comparison  form  for  each  year 
in  the  first  solicitation  period  do  not  actually  correspond  to  what  was 
spent  on  an  annual  basis.  Changes  in  the  demand  for  a  particular 
function,  as  well  as  changes  in  which  tasks  should  be  conducted,  are 
constantly  changing.  However,  these  changes  are  not  part  of  the  A-76 
process.  These  changes  would  have  occurred  whether  or  not  the 
function  was  ever  competed.  By  isolating  the  components  of  total 
costs,  we  can  track  cost  increases  or  decreases  and  determine  whether 
changes  would  have  occurred  if  the  competition  had  never  been  con¬ 
ducted.  Therefore,  to  evaluate  contract  costs,  we  looked  at  the  funds 
available  for  each  year  of  the  contract  and  tracked  the  modifications 
made  during  the  year.  For  in-house  wins,  we  tracked  annual  costs 
from  budget  and  manning  documents  with  estimates  for  the  impact 
of  changes  during  the  year.  Each  of  the  components  that  impact 
annual  cost  are  outlined  below: 

•  Scope  changes.  Scope  changes  are  defined  as  increases  or 
decreases  in  the  set  of  functions  defined  in  the  PWS.  Examples 
include  adding  facility  painting  to  a  maintenance  contract  that 
was  limited  to  minor  repair,  thereby  eliminating  the  self-help 
stores  from  a  supply  contract.  If  a  scope  change  occurs  in  the 
first  year  of  a  contract,  the  funds  made  available  for  all  subse¬ 
quent  years  of  the  contract  can  reflect  the  increased  or 
decreased  scope. 

•  Workload  changes.  Workload  changes  are  defined  as  the  increase 
or  decrease  in  the  level  of  effort  required  under  the  PWS. 
Examples  include  increasing  the  number  of  improved  acres  in 
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a  grounds  maintenance  contract  or  decreasing  the  number  of 
passenger  vehicles  maintained  in  a  vehicle  maintenance  con¬ 
tract.  After  a  workload  adjustment  occurs,  the  funds  made  avail¬ 
able  for  all  subsequent  years  of  the  contract  can  reflect  the 
impact  of  the  workload  change. 

One-time  cost  changes.  One-time  cost  adjustments  are  defined  as 
a  scope  or  workload  increase  that  impacts  only  the  current  year 
of  the  contract.  For  example,  one  installation  suffered  signifi¬ 
cant  damage  from  a  hurricane.  The  facility  maintenance  and 
grounds  maintenance  contracts  at  this  installation  were  modi¬ 
fied  to  increase  the  workload,  or  level  of  effort,  to  clean  up  the 
debris  and  re-build.  However,  the  funds  available  for  subse¬ 
quent  years  were  not  increased  because  the  workload  was  not 
expected  to  continue. 

Wage  determinations.  Throughout  the  period  of  performance, 
Department  of  Tabor  or  Service  Contract  Act  wage  determina¬ 
tions  will  increase  labor  rates  and,  therefore,  the  annual  cost  of 
the  contract  above  what  was  originally  bid.  Wage  determina¬ 
tions  will  affect  not  only  the  current  year  of  the  contract  but  all 
subsequent  years  as  well.  In  the  case  of  in-house  wins,  wage 
increases  are  also  calculated. 

Cost  adjustments.  For  contracts  that  include  labor  and  materials, 
there  can  be  adjustments  to  the  unit  cost  of  materials  as  the 
contract  period  moves  forward.  For  example,  increases  in  fuel 
costs  will  increase  the  costs  of  performing  shuttle  services 
under  a  vehicle  operations  contract.  This  type  of  cost  increase 
would  affect  the  cost  of  providing  this  function  whether  or  not 
the  function  was  performed  via  in-house  or  contract  workforce. 
We  have  assumed  that  a  unit  price  increase  or  decrease  will  con¬ 
tinue  throughout  the  contract  period. 

Labor  augmentation.  For  certain  contracts,  particularly  those 
with  significant  performance  problems,  government  labor  may 
be  used  to  augment  the  provision  of  contract  services.  Through 
interviews  with  customers  and  management,  we  estimated  the 
size  of  this  workforce  and  developed  fully  burdened  rates  for 
this  labor  on  an  annual  basis.  In  other  cases,  the  total  number 
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of  personnel  involved  with  management  of  the  function  was 
larger  than  estimated  in  the  cost  comparison.  Through  inter¬ 
views,  we  also  estimated  the  fully  burdened  cost  of  this  additional 
labor  force. 

Comparing  costs  and  savings 

The  annual  costs  of  a  function  are  the  funds  made  available  to  conduct 
that  function  at  the  start  of  the  fiscal  year,  plus  or  minus  adjustments 
made  during  the  year  and,  for  contract  wins,  the  total  costs  to  the  gov¬ 
ernment  from  contract  administration  and  management  including 
QAEs.  These  are  the  annual  costs  observed  by  the  government  for  the 
provision  of  the  function. 

To  determine  how  these  costs  compare  with  what  was  “bid”  in  the  cost 
comparison  relative  to  the  original  PWS,  adjustments  for  changes  in 
workload,  scope,  unit  cost  and  wage  changes  must  be  accounted  for  not 
only  in  the  year  in  which  they  occurred  but  for  all  subsequent  years  as 
well.  To  this  end,  if  a  scope  increase  of  $50,000  occurs  in  a  given  year,  it 
is  expected  that  the  funds  made  available  for  all  remaining  years  will  be 
$50,000  higher  than  what  is  projected  in  the  cost  comparison  form.1 
This  increase  in  cost  reflects  the  provision  of  additional  effort,  not  an 
increase  in  the  cost  of  providing  the  original  functions  defined  in  the 
PWS.  Therefore,  to  ensure  an  apples-to-apples  comparison  of  the  cost 
of  providing  the  original  set  of  functions,  the  $50,000  for  additional 
workload  would  be  subtracted  out  of  the  funds  available  for  each 
subsequent  year  of  the  contract. 

To  determine  the  savings  achieved  for  the  16  competitions,  we  evalu¬ 
ated  and  compared  costs  and  savings  from  three  perspectives:  expected, 
observed,  and  effective.  Using  this  approach  allows  us  to  separate  and  eval¬ 
uate  the  costs  of  meeting  the  tasks  described  in  the  original  PWS,  and 
the  impact  on  costs  from  changes  in  scope,  workload,  and  other  adjust¬ 
ments. 


1.  Adjustments  that  are  made  mid-year  are  annualized  for  all  subsequent 
years  (e.g.  it  is  assumed  that  a  $20,000  scope  change  impacting  6  months 
of  a  given  year  will  impact  each  subsequent  year  by  $40,000) . 
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Definition  of  terms  and  method  of  calculation 

Definitions  of  the  terms  we  used  and  method  of  calculation  follow: 

•  Expected  costs  are  defined  as  what  the  government  expects  to  pay 
for  the  provision  of  a  commercial  function  after  a  competition 
is  completed  (e.g.,  the  price  of  the  winning  bid  plus  all  admin¬ 
istrative  costs  to  the  government) .  Expected  savings  are  estimated 
as  the  difference  between  what  the  government  expects  to  pay 
and  the  pre-competition  costs  of  providing  the  function. 
Expected  costs  and  savings  are  forecasts  based  on  the  winning 
contract  or  MEO  bid  at  the  time  of  competition  and  can  be 
incorporated  into  out-year  budget  decisions. 

The  expected  cost  is  given  by  the  following  formula 

XCj  =  Cf+  Ap  where 

XC  =  Expected  costs.  The  annual  costs  the  govern  men  t  expects  to  pay  for  a  given  year 
C  =  The  total  winning  contract  bid  (or  MEO)  for  a  given  year 
A  =  The  total  administrative  and  other  costs  to  the  government  as  reported  on  the  cost 
comparison  form  for  a  given  year.  (Note:  one-time  conversion  costs  are  annualized 
across  the  first  solicitation  period). 

•  Observed  costs  are  defined  as  what  was  actually  spent  by  DoD  for 
the  provision  of  services.  Observed  costs  include  increases  or 
decreases  to  annual  costs  from  changes  in  scope,  workload, 
wages,  and  one-time  cost  adjustments.  Observed  savings  are  the 
difference  between  the  pre-competition  annual  cost  to  the  gov¬ 
ernment  and  the  actual  or  observed  costs  of  that  function  after 
the  competition  was  completed. 

The  observed  cost  is  given  by  the  following  formula: 

OC j  —  F ^  +  Aj  +  S ^  +  I) ^  +  0^  +  W t  +  P ^  +  E where 

OC  =  Observed  Cost.  The  annual  cost  the  government  is  required  to  pay  for  a  given  year 
F=  Actual  funds  made  available  for  a  given  contract  or  MEO  at  the  beginning  of  a 
given  fiscal  year 

A  =  The  total  administrative  and  other  costs  to  the  government  as  reported  on  the  cost 
comparison  form  for  a  given  year.  (Note:  conversion  costs  are  annualized  across  the  first 
solicitation  period). 
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S  =  Total  annual  increase  or  decrease  in  cost  due  to  scope  changes  for  a  given  year. 

D  =  Total  annual  increase  or  decrease  in  cost  due  to  workload  changes  for  a  given  year 
O  =  Total  annual  increase  in  cost  due  to  one-time  cost  changes  for  a  given  year. 

W  =  Total  ann  ual  increase  or  decrease  in  cost  due  to  periodic  changes  in  wage  rates 
proscribed  by  the  Department  of  Labor  or  the  Service  Contract  Act. 

P  =  Total  ann  ual  increase  or  decrease  in  cost  due  to  changes  in  the  unit  cost  of  materi¬ 
als. 

L  =  Total  annual  increase  in  cost  due  to  labor  augmentation  for  a  given  year. 


•  Effective  costs  are  defined  as  the  estimated  cost  to  DoD  of  provid¬ 
ing  the  same  set  of  services  as  originally  identified  in  the  cost 
comparison.  Effective  cost  estimates  exclude  cost  changes  that 
would  have  occurred  whether  or  not  the  function  was  com¬ 
peted.  For  example,  in  one  competition  the  observed  costs  of 
providing  services  increased  by  over  15  percent  from  1991  to 
1992.  This  increase  was  due  to  additional  workload  needed  to 
support  our  military  in  the  Persian  Gulf.  This  type  of  increase 
in  workload,  and  therefore  cost,  would  have  occurred  whether 
the  necessary  services  were  provided  via  contract  or  through  in- 
house  labor.  Therefore,  the  effective  costs  for  1992  would  be 
adjusted  to  remove  these  one-time  costs.  By  adjusting  the  data 
to  exclude  workload,  scope,  wage,  and  one-time  costs,  effective 
cost  estimates  allow  us  to  compare  changes  in  cost  while  keep¬ 
ing  the  original  scope  constant.  Effective  savings  are  defined  as 
the  difference  between  the  pre-competition  annual  cost  to  the 
government  and  the  effective  cost  that  function  after  adjust¬ 
ments  are  made.  Comparing  effective  and  pre-competition  cost 
estimates  provides  insight  into  true  cost  growth  or  savings. 

Based  on  the  observed  cost  and  expected  cost  formulas  above, 
the  effective  cost  is  given  by  the  following  formula: 


t 

EC,  =  ocrl.ls,+D,+  w,+pi>vol*Ll) 

i=  1 
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where  the  effective  cost  (EC)  for  a  given  year  is  equal  to  the  observed  costs  of  the  function 
less  the  cumulative  impact  of  all  scope,  workload,  ivage  and  price  adjustments  occur¬ 
ring  since  contract  inception  and  less  the  one  time  cost  changes  and  labor  augmenta¬ 
tion  for  the  year  of  calcula  tion  only. 

Effective  costs  are  the  most  meaningful  indication  of  whether  an  A-76 
competition  was  successful  in  producing  real  and  sustained  savings 
because  they  identify  the  costs  of  providing  the  same  scope  of  work 
over  time.  However,  it  is  also  important  to  examine  changes  in 
observed  costs  because,  historically,  these  are  the  types  of  costs  people 
have  looked  at  when  examining  the  worth  of  the  competitive 
sourcing  program. 

Caveats  and  assumptions 

During  our  analysis,  we  had  to  make  some  assumptions  in  isolating 
such  factors  as  the  effects  of  scope  or  workload  changes,  the  amount 
of  contract  administration  or  augmentation  of  contract  labor  by  gov¬ 
ernment  labor,  or  minor  discrepancies  between  authorized  and 
expended  funds.  In  all  cases,  we  chose  to  be  conservative  and  decided 
in  favor  of  the  alternative  that  would  limit  rather  than  increase 
savings. 

Scope  and  workload  changes  versus  one-time  cost  increases 

It  was  sometimes  difficult  to  determine  the  difference  between  these 
two  changes.  When  in  doubt,  we  tried  to  guess  conservatively — decid¬ 
ing  in  favor  of  a  scope  change  rather  than  a  one-time  change.  How¬ 
ever,  if  it  was  in  fact  a  one-time  change,  then  future  years  adjusted 
(effective)  costs  will  be  lower  than  we  show  them. 

Baseline  costs 

If  the  baseline  FTE  billet  estimates  provided  by  the  Services  are  for  a 
different  set  of  functions  than  was  in  the  PWS,  baseline  estimates  will 
be  wrong.  This  will  impact  savings  estimates  provided  later  in  this 
study.  We  have  assumed  baseline  billets  are  correct  and  for  the  same 
set  of  functions  in  the  PWS. 
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Labor  augmentation 

These  estimates  were  based  on  estimates  provided  to  us  during  our  in 
interviews  with  relevant  personnel  and  customers.  To  our  knowledge, 
there  are  no  documented  data  in  this  area. 

Annualizing  costs 

If  there  is  a  modification  to  a  contract  or  the  MEO  that  occurs  6 
months  into  the  solicitation  period,  then  we  doubled  the  cost  of  the 
modification  for  subsequent  years.  However,  certain  modifications 
occur  mid-year  but  actually  cover  the  full  year.  We  have  tried  to  be 
conservative  in  our  estimates,  and  identify  as  many  of  these  situations 
as  possible.  A  good  example  of  this  type  of  modification  is  in  the 
grounds  maintenance  area.  If  a  scope  change  occurs  in  January,  well 
before  the  growing  season,  the  cost  change  is  most  likely  for  the 
whole  year,  and  we  have  treated  it  as  such. 

In-house  wins 

Because  changes  in  staffing  typically  were  not  tied  to  specific  changes 
in  scope  or  workload,  it  was  not  possible  to  evaluate  observed  or  effective 
costs  and  savings  for  the  in-house  wins  in  our  sample. 

Wage  changes 

We  have  assumed  wage  changes  to  contract  labor  would  have 
occurred  at  similar  levels.  Although  we  would  have  preferred  to  eval¬ 
uate  the  difference  in  wages  between  contract  and  in-house  work¬ 
force,  there  were  many  instances  where  wage  determinations  were 
coupled  with  other  scope  or  workload  adjustments.  Therefore,  we 
could  not  isolate  changes  in  wages  from  the  impact  of  other  changes. 
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Appendix  D:  A  case  study 


As  we  mentioned  at  the  beginning  of  this  section,  there  was  one  com¬ 
petition  for  which  many  of  the  officials  we  interviewed  declined  to 
rate  the  provider’s  performance.  This  competition  covered  all  the 
major  functions  at  the  installation  and  was  extremely  large.  At  the 
time  of  our  review,  they  were  in  delicate  and  ongoing  contract  nego¬ 
tiations.  As  a  result,  the  officials  were  uncomfortable  providing  rat¬ 
ings  that  might  be  different  than  the  official  performance 
evaluations.  For  this  reason,  we  excluded  it  from  the  analysis  above, 
and  are  examining  it  separately. 

According  to  official  quarterly  performance  evaluations  at  the  time  of 
our  examination,  the  contractor  is  performing  17  out  of  a  total  of  21 
functions,  or  81  percent,  at  a  good  to  excellent  level.  However,  this 
seemingly  good  rating  belies  the  level  of  tension,  frustration,  and  dis¬ 
satisfaction  that  both  government  and  contract  representatives  feel. 
The  competition  has  been  plagued  from  the  beginning  with  a 
number  of  difficulties,  some  of  which  the  installation  is  trying  to  over¬ 
come  over  a  year  after  the  start  of  the  contract  performance.  The 
principal  difficulties  include: 

•  Change  in  mission 

•  Poor  workload  data 

•  Differing  expectations  between  the  contractor  and  base  per¬ 
sonnel 

•  Lack  of  sufficient  experienced  people  to  perform  all  of  the 
functions 

•  Short  transition  period. 

During  the  competition,  the  prevailing  view  was  that  the  installation’s 
importance  to  the  Service,  and  thus  its  workload,  would  continue  to 
diminish  over  time  and  that  the  installation  might  eventually  be 
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closed.  Potential  contractors,  for  example,  were  asked  to  address  how 
they  would  scale  down  operations  in  their  proposals.  As  a  conse¬ 
quence,  the  PWS  requirements  in  the  solicitation  were  less  than  those 
being  provided  at  the  installation  during  the  competition.  Shortly 
after  the  decision  was  made  to  convert  the  functions  to  contract,  the 
installation’s  strategic  importance  to  the  Service  was  increased  signif¬ 
icantly.  Instead  of  eventual  closure,  the  installation  has  been  given 
additional  mission  responsibilities  and  workload  has  increased  sub¬ 
stantially.  There  was  no  corresponding  adjustment  to  the  contract, 
and  the  fixed  price  nature  of  the  contract  makes  it  difficult  to  do  so. 

The  change  in  strategic  importance  was  exacerbated  by  two  addi¬ 
tional  factors — poor  workload  data  in  the  PWS  and  differing  expecta¬ 
tions  of  headquarters,  installation,  and  contract  personnel.  Once  the 
transition  began,  the  contractor  found  that  the  workload  data  on 
which  it  had  based  its  proposal  were  incomplete,  inaccurate,  and  out¬ 
dated.  Poor  workload  data  combined  with  a  changing  mission  made 
it  difficult  for  the  contractor  to  accurately  predict  what  needed  to  be 
done. 

The  competition  was  conducted  by  headquarters  personnel.  Because 
the  prevailing  view  was  that  the  installation  was  likely  to  close,  they 
developed  a  PWS  with  a  lower  level  of  service  than  was  currently  being 
performed.  The  headquarters  personnel  did  not  effectively  commu¬ 
nicating  these  differences  to  the  installation  personnel  who  were  to 
manage  the  functions  after  the  competition  was  completed.  The 
installation  personnel  expected  to  continue  the  same  level  of  perfor¬ 
mance,  and  in  much  the  same  manner  as  it  had  prior  to  competition. 
These  different  levels  of  expectation  made  the  transition  difficult  and 
created  tensions  between  the  installation  personnel  and  the  contrac¬ 
tor  that  continue  to  persist.  As  a  consequence,  there  is  a  tendency  for 
installation  personnel  to  expect  a  performance  level  that  is  as  good 
or  better  than  it  was  before  competition,  while  the  contractor  is 
strictly  and  narrowly  interpreting  the  performance  requirements  in 
the  contract  in  an  attempt  to  keep  costs  down. 

The  winning  contractor  had  90  days  to  make  the  transition  to  full  per¬ 
formance.  During  that  time,  the  contractor  had  to  interview  employ¬ 
ees  and  make  job  offers,  familiarize  themselves  with  the  installation 
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and  the  operations  that  they  were  taking  over,  and  to  inventory  the  equip¬ 
ment  that  was  being  transferred  to  the  contractor.  Because  a  large 
number  of  employees  accepted  priority  placement  program  offers  for 
jobs  far  from  the  installation  and  the  local  labor  market  did  not  have 
many  skilled  workers,  the  contractor  found  it  very  difficult  to  hire  suffi¬ 
cient  numbers  of  people.  As  a  consequence,  it  spent  the  majority  of  the 
transition  period  trying  to  hire  employees  and  was  not  able  to  adequately 
shadow  the  government  operations  or  meet  its  own  milestones.  The  con¬ 
tractor  has  had  to  bring  some  of  his  employees  from  other  locations  to 
make  up  for  the  lack  of  qualified  personnel  locally.  Many  of  these 
employees  were  sent  on  a  temporary  basis,  but  are  still  in  place  over  a  year 
later. 

The  inability  to  hire  sufficient  numbers  of  experienced  government  per¬ 
sonnel  denied  the  contractor  access  to  historic  information  and  sharply 
increased  his  learning  curve.  The  contractor’s  representatives  stated  that 
a  6-month  transition  period  would  have  been  more  appropriate  consid¬ 
ering  that  the  competition  encompassed  more  than  20  functions.  The 
installation’s  contract  personnel  agree  that  a  longer  transition  time 
would  have  helped  the  contractor,  but  believe  that  the  customers  would 
have  been  very  unhappy. 

Although  the  contractor  was  willing  to  increase  his  level  of  support  to 
meet  the  new  requirements  and  increased  workload,  he  believes  that  he 
cannot  accomplish  it  within  the  scope  of  the  current  fixed-price  contract. 
At  the  time  of  our  review,  both  the  Service  and  the  contractor  were 
addressing  the  changes  needed  to  clarify  performance  requirements  and 
reflect  the  differences  in  expectations,  and  expected  to  make  the 
necessary  contract  modifications. 
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